- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how language models respect role-specific cognitive boundaries during role-playing. The authors construct a counterfactual dataset based on occupational knowledge and employ activation patching, causal mediation, and representation steering to analyze the underlying process. The paper proposes a four-stage mechanism consisting of role encoding, query conditioning, conflict signal formation, and response action binding, with experiments on Qwen, Llama, and TimeChara.
Critical strengths
The causal experimental design is relatively systematic. Beyond standard activation patching, the authors use Query clamp/replay to investigate information propagation and introduce cross-role transfer and response-mapping experiments to distinguish internal functions. The results also show a degree of consistency across multiple models and an external benchmark.
Critical weaknesses
My main concern is the definition of cognitive boundaries. The authors essentially treat occupation-specific knowledge requirements as the boundaries of what a role can know, assuming that questions outside the associated professional domain should be withheld. However, occupational expertise and individual knowledge boundaries are not equivalent. For example, an archive manager may still know the definition of ROA. A professional role does not necessarily determine what an individual knows or does not know. Consequently, the experiments may primarily capture the model's sensitivity to occupational relevance and role instructions, rather than genuine cognitive-boundary reasoning. This distinction fundamentally affects the interpretation of the proposed mechanism.
Questions for authors
Figures 5 and 6 present the most important results, but I am not fully convinced by the interpretation of the shared conflict signal. The authors construct representation differences between role-query conditions and show that a particular component can induce Unknown across different roles and queries. However, this does not sufficiently establish that the component specifically encodes cognitive-boundary conflict. I would expect more direct evidence of the signal's semantic function rather than interpreting its ability to induce withholding as sufficient evidence of a conflict representation.
The paper proposes a four-stage mechanism: Role Encoding, Query Conditioning, Conflict Formation, and Response Action Binding. However, I think the first two stages may partly reflect the basic computational structure of autoregressive Transformers. Since role information precedes the query, the Query position can naturally access that information, while the Response position can access both.
Where are your contributions in sec 1?
6 comments
Clarification of Our Mechanistic Account
We thank the reviewer for their questions about our mechanistic account. Before addressing these questions point by point, we would first like to clarify this account further, so that our contribution can be better understood.
For ease of reference to the paper, we follow the causal analysis framework to explain this mechanism.
-
Section 4.1: Causal Structure Identification
In Section 4.1, we identify the causal structure through which the model constrains its use of broad knowledge. This structure begins at Role, within a window in the early layers. Within this window, Role is in a state that causally constrains behavior; that is, its representation contains causal components that produce this constraint. Within a middle-layer window, the causal effect induced by Role propagates to Query. Under this influence, Query forms a conditioned state that also has a causal effect. Although the autoregressive architecture allows Role to causally affect both Query and Response, our mediation experiments show that Query mediates almost the entire causal effect of Role. This implies a strictly ordered propagation structure: Role’s causal components affect how the Query state forms, and Query in turn affects how the Response state forms. Finally, the causal components at Response are decoded into concrete, constrained behavior from the role’s cognitive perspective.
We emphasize that, although this causal structure is consistent with the standard autoregressive architecture, it explains which positions and layers have causal effects and how these effects propagate. It also resolves the uncertainty left by the standard architecture. This structure provides the basis for our subsequent causal analysis.
-
Section 4.2: Causal Computation Identification
Within the identified causal structure, we further identify the computational mechanism that translates the causal effect into concrete, constrained behavior. We find that the concrete response forms mainly at later Response layers, whereas earlier Response representations can already constrain the model’s use of broad knowledge.
We further examine how this earlier causal effect relates to the concrete response and identify a transformation window. Across this window, the earlier causal effect gradually weakens as the concrete response forms. Together, these findings reveal how the earlier causal effect is translated into a concrete response.
Within this computational mechanism, we further identify and interpret a causal component underlying the earlier effect. It arises from the joint effect of the role’s cognitive boundary and the current query when they conflict. The component is decoupled from the specific role and query, as well as from the subsequent concrete response. It transfers across contexts and still causally constrains the model’s use of broad knowledge. We call it the conflict signal.
-
Section 4.3: Causal Computation Reconstruction
Having identified this computation, we follow the established causal structure to reconstruct the model’s full computational process. Since Query mediates almost the entire causal effect, we first examine what attention at Response writes when reading Query and how these writes form the conflict signal. We find that attention at Response already writes the conflict signal into the Response representation when reading the role-conditioned Query state. Writes across layers jointly reinforce the signal.
This result further shows that, as Query mediates the role-induced effect, the current query has already been conditioned by the role’s cognitive boundary. This forms a role–query state for downstream use. Response uses this state to constrain subsequent knowledge use through the conflict signal.
We then follow the same causal structure upstream to identify which Role components causally affect subsequent Query processing. We find that the effect mediated by Query is carried mainly by components formed during forward encoding of the role profile. The initial embedding component has little effect.
These findings further show that how the model implements the role’s cognitive boundary depends on how it encodes the profile into an internal role state. They also suggest a mechanistic explanation for LMs’ sensitivity to profile wording. Even when descriptions refer to the same role, different wording may change the internal role state the model forms. Through the downstream causal pathway, these changes may in turn affect how the cognitive boundary constrains responses.
Together, these analyses reveal the full causal process through which LMs interpret the role’s cognitive boundary, assess whether the current query conflicts with it, and use the conflict signal to constrain what they present as known.
Response to Questions (Part 1/2)
We thank the reviewer for their three specific questions about our mechanistic account. Below, we address the reviewer’s concerns point by point.
Question 1: The authors show that a particular component can induce Unknown across roles and queries, but this does not sufficiently establish that it specifically encodes cognitive-boundary conflict. Is there more direct evidence of the signal’s semantic function beyond inducing withholding?
First, we would like to clarify that all our conclusions serve to reconstruct the mechanism by which the model constrains its use of broad knowledge to respect the role’s cognitive boundary. In particular, we interpret the conflict signal as a component produced by the joint effect of the query and the role’s cognitive boundary when they conflict. The model uses this component to mark the effect induced by the conflict: constraining its use of the broad knowledge needed to answer the query. We do not claim that the conflict signal directly encodes conflict unrelated to behavioral constraints.
Second, we would like to clarify that the conflict signal is not limited to inducing “Unknown.” In our experiments, we prescribe “Unknown” as one way for the model to express the role’s lack of knowledge. In the response-mapping and cross-mapping experiments in Section 4.2 (details in Appendix D.5), we prescribe different forms of expression and further demonstrate that the conflict signal is decoupled from this specific response behavior.
We next present direct experimental evidence that the model uses the conflict signal to mark the behavioral constraint induced by conflict.
First, our causal experiments strictly control the variables. We control the model’s behavioral differences so that they arise only from its judgment of whether the query conflicts with the role’s cognitive boundary. Thus, the causal effect in the experiment is induced solely by a change in the role’s cognitive boundary. This ensures that our analysis is aligned with the cognitive-boundary task. Second, we establish that the earlier causal effect at Response is decoupled from concrete behavior. It is an abstract effect that constrains model behavior but is not tied to a specific response.
On this basis, there are two possible explanations for the earlier causal effect. (1) The model has not yet completed its assessment of the query against the role’s cognitive boundary. The representation retains separate role and query effects, which act together here to constrain the concrete response. (2) The model has already completed this assessment. The representation retains their joint effect, which constrains the concrete response.
To understand this conclusion, we start with the experimental setup. A reciprocal family contains two roles, , and their corresponding queries, . When a role is paired with its corresponding query (e.g., with ), the model judges that there is no conflict; otherwise, it judges that there is conflict. With query fixed, we subtract the Response representation under from that under to obtain . This cancels the separate effect of the shared query. Thus, retains the difference between the effects of and , together with the joint effect of the query and role. For query , we subtract the representation under from that under to obtain . It retains the difference between the effects of and , together with the joint effect of the query and role. We then form:
Here, cancels the separate role effect and retains the joint effect of the query and role; does the reverse, retaining the separate role effect and canceling the joint effect. We provide the detailed derivation in Appendix D.6. In Figure 5, we also include a random-perturbation control. The effects of and the random perturbations are almost equally low, whereas remains highly effective. This demonstrates that the causal effect in this component arises from the joint contribution of the role and query. Of course, this does not yet establish that the component is generalizable. The key evidence is that we apply and to a different reciprocal family, where both the roles and queries differ. Figure 5 shows that the effect of cross-family is almost identical to that of same-family . We have thus identified a set of components induced by conflict between the query and the role’s cognitive boundary. They are decoupled from both, as well as from the concrete response, yet they causally constrain the model’s behavior. We call this the conflict signal.
Response to Weakness
We thank the reviewer for reviewing our work and for their thoughtful comments.
We would first like to clarify that we do not study what a real individual knows or does not know. We place our study strictly within the context of role-playing. This requires the model to first understand the role specified in the instructions and then respond consistently with that role. The model must not only imitate the role’s style or personality but also reflect its cognitive perspective in its behavior. We therefore view a cognitive boundary as a role attribute, like style or personality. The model must interpret this attribute and consistently reflect it in its behavior.
Second, we would like to clarify why we use occupational knowledge. To study this attribute, we need the model to exhibit two contrasting behaviors under different roles: expressing knowledge and expressing a lack of knowledge. Highly specialized occupations have relatively clear links to specific knowledge. We use these occupations as the basis for role-playing to more readily elicit different cognitive boundaries for the same query. The model is then more likely to interpret a specialized query as within a related role’s cognitive boundary and beyond an unrelated role’s boundary. We thus form those counterfactual pairs for causal analysis.
For example, given a specialized financial question, a model may consider a banking adviser more likely to know the answer and an archive manager less likely to know it.
We emphasize that we do not assume occupational relevance or irrelevance will necessarily yield opposite boundary judgments. During role-playing, a role’s cognitive boundary depends on the model’s interpretation. The same role–query pair may therefore lead to different judgments across models. For example, a Qwen model may judge that the archive manager cannot answer that financial question, while a Llama model may reach the opposite judgment. We therefore screen samples separately for each model, retaining role–query pairs for which its boundary judgments match our predefined expectations. This grounds our subsequent analysis in the boundary judgments actually expressed by each model.
Together, we use occupational role profiles to control how the model interprets cognitive boundaries as a role attribute. On this basis, we study how this attribute constrains the model’s use of broad knowledge so that it responds from the role’s narrower cognitive perspective.