acceptodds
Under review as a conference paper at ICLR 2027

LLMs Already Know What Not to Share: Span-Level Contextual Integrity via Probing and Attention Masking

14 views

Abstract

As large language model (LLM) agents act on users' behalf, they are increasingly given access to personal information. Yet access does not imply permission to disclose: under Contextual Integrity (CI), the appropriateness of each information span depends on the current task. Given their large-scale training, we hypothesize that LLMs possess an implicit capacity for CI, reflected in contextual privacy signals encoded during prompt processing. To formalize this capacity, we propose the Contextual Linear Privacy Hypothesis, which models privacy-relevant information as a linear component of the span representation whose strength varies with context, and characterizes how changing context shifts the representation of a fixed span. We operationalize this hypothesis with layer-specific linear probes and introduce probe-guided attention masking, which prevents generated tokens from attending to cached positions classified as restricted without modifying the base model or generating additional privacy reasoning. Across five models, probes average accuracy and move in the expected direction on of 226 context-flip pairs. Our masking achieves the highest Completeness on every model ( versus for CI-CoT), while averaging 178 generated tokens compared with 475 for CI-CoT. These results show that contextual privacy signals in LLM representations can support targeted and efficient disclosure control.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Contextual Integrity, Inference-time Privacy, Probing, Large language models, Attention Steering

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_mGSbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This research investigates whether LLMs encode context-dependent privacy information in their internal representations and whether such signals can be used to prevent inappropriate information disclosure and proposes the Contextual Linear Privacy Hypothesis and introduce a probe-guided attention masking framework. Specifically, a linear probe identifies restricted information spans from contextualized hidden representations, and the corresponding KV-cache positions are masked during generation. Experiments across five LLMs demonstrate promising results in contextual privacy classification and disclosure control.

Critical strengths
  1. This research addresses an important and practical problem by considering both training-time and inference-time privacy within a unified framework.
  2. The proposed framework is reasonable and relatively lightweight. Combining span-level linear probing with attention masking provides a straightforward approach to controlling contextual information disclosure without modifying the base model.
  3. The presentation and organization are generally strong.
Critical weaknesses
  1. Limited evidence supporting the Contextual Linear Privacy Hypothesis. Although the high accuracy of linear probes suggests that contextual privacy information is linearly accessible from hidden representations, this does not necessarily establish that LLMs intrinsically recognize privacy boundaries or just rely on such representations when generating responses. The proposed decomposition also does not uniquely identify a privacy-specific component. Therefore, the relationship between linear decodability and the claimed internal privacy awareness could be more carefully justified.
  2. Limited generalization to realistic and unstructured scenarios. The experiments rely on a single synthetic contextual integrity benchmark with structured information fields and predefined span boundaries. However, real-world LLM agents frequently process unstructured documents, conversational histories, and information whose privacy implications depend on multiple interacting spans. It remains unclear how the proposed method would generalize to such scenarios or how candidate spans would be reliably identified in practice.
  3. Limited evaluation of semantic information leakage. The Integrity metric may not fully capture semantic information leakage. For example, if the restricted information is "Annual salary: $120,000," the model might disclose "The employee earns approximately 120 thousand dollars per year" without reproducing the original value verbatim. Would such semantically equivalent disclosures be considered privacy violations under the current evaluation? Clarifying this would strengthen the reliability of the reported privacy protection results.
Questions for authors
  1. Could the authors further clarify whether the Contextual Linear Privacy Hypothesis implies only linear decodability or also suggests a causal role of privacy-related representations in generation?
  2. How would the proposed framework handle unstructured input, where sensitive spans are not explicitly identified or may overlap with other information?
  3. Does the Integrity evaluation account for semantic disclosures, including paraphrases or partial reconstruction of restricted information? as example shown in W3
2 comments

We thank the reviewer for the careful reading and the constructive comments.

W1/Q1. Limited evidence supporting the Contextual Linear Privacy Hypothesis.

Our hypothesis concerns the linear decodability of contextual disclosure labels from prefill span representations. Equation (2) provides a modeling decomposition. Whether the unmodified model uses this signal during generation is a separate causal question [1].

In our ablation (Fig. 2a and Appendix E.1), we estimated a steering direction from the difference between the allowed and restricted span centroids in prefill. We added it to one layer’s residual stream at every prompt position and decoding step. Across five models, steering achieved 12.6% and 14.8% mean Completeness at strengths of 0.5 and 1.0, respectively, close to Base (14.5%). Probe-guided attention masking achieved 32.3%.

Prefill–decode mismatch is a recurring challenge for activation steering because a direction estimated from prefill activations may transfer poorly to decode-time states [2]. Our attention-masking design sidesteps the prefill–decode transfer challenge by using the privacy signal read during prefill to guide a span-specific intervention during generation.

W2/Q2. Limited generalization to realistic and unstructured scenarios.

Please see our response to Reviewer 95oh (W2a) for how we interpret the 96.2% directional consistency and what the contextual-shift results support.

W3/Q3. Limited evaluation of semantic information leakage.

We thank the reviewer for suggesting that we check for semantic privacy leakage. Our original Table 3 followed the synthetic-benchmark evaluation pipeline of Lan et al. [3], which checks the final answer for annotated restricted keywords when scoring Integrity. Both the benchmark paper’s evaluation on this dataset and our original results left coverage of paraphrases and partial reconstructions unverified.

We re-evaluated the existing final answers underlying Table 3 using human review cross-checked with GPT-6.1 Sol. We checked whether each answer conveyed restricted information through rewording or made it recoverable in context.

The review identified previously uncounted semantic disclosures in 6 Base outputs, 14 CI-CoT outputs, and 3 Masking outputs, out of 365 outputs per method. For example, Mistral-7B disclosed a long-standing desire to buy a Tesla by saying the person had “have been eyeing this particular model for quite some time.”

We counted these outputs as Integrity violations and recomputed Completeness. Under the expanded evaluation, Masking achieved the highest Integrity and Completeness on all five models. We will document the semantic review procedure in the main text and update Table 3.

ModelBase I↑Base U↑Base C↑Base Cost↓CI-CoT I↑CI-CoT U↑CI-CoT C↑CI-CoT Cost↓Masking I↑Masking U↑Masking C↑Masking Cost↓
Qwen2.5-7B35.664.420.516032.961.619.247272.653.435.6151
Qwen2.5-14B27.460.317.816845.256.221.955972.648.034.2161
Llama-3.1-8B23.368.511.018954.857.524.754365.849.331.5217
Mistral-7B5.571.22.723334.267.121.945349.361.628.8241
Gemma-2-9B52.142.515.112354.847.920.534975.339.728.8117
Average28.861.413.417544.458.121.647567.150.431.8178

References

[1] Shao, Z., et al. (2026). DecodeShare: Tracing the Shared Subspace of LLM Decode-Time Decisions. Proceedings of the 43rd International Conference on Machine Learning. Paper

[2] Elazar, Y., Ravfogel, S., Jacovi, A., & Goldberg, Y. (2021). Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals. Transactions of the Association for Computational Linguistics, 9, 160–175. Paper

[3] Lan, G., et al. (2025). Contextual Integrity in LLMs via Reasoning and Reinforcement Learning. NeurIPS 2025. See §4 for the benchmark’s Integrity and Completeness definitions.

Thank you again for your valuable comments and constructive suggestions. We have carefully addressed your concerns in our rebuttal and provided additional clarifications where necessary. As the discussion period is limited, we would be most grateful if you could find a moment to look over our response and let us know whether our explanations have adequately addressed your questions. Should any concerns remain, we would be very happy to provide further clarification. Since our experiments are computationally intensive and take considerable time to complete, an early reply would be of great help in allowing us to prepare any additional results you may wish to see within the remaining discussion period. We sincerely appreciate your time and effort in reviewing our paper.

Reviewer_vUUqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether LLMs internally encode context-dependent privacy judgments and whether these signals can be leveraged to prevent inappropriate information disclosure during generation. The authors propose the Contextual Linear Privacy Hypothesis, which models the representation of a span SS under context CC as

hS(ℓ)(C)=hbase,S(ℓ)(C)+αS(ℓ)(C)vP(ℓ)+ϵS(ℓ)(C).\mathbf{h} _S^{(\ell)}(C) = \mathbf{h} _{\mathrm{base},S}^{(\ell)}(C) + \alpha _S^{(\ell)}(C)\mathbf{v} _P^{(\ell)} + \boldsymbol{\epsilon} _S^{(\ell)}(C).

Here, vP(ℓ)\mathbf{v} _P^{(\ell)} represents a hypothesized privacy-related direction whose coefficient varies with context. To operationalize this hypothesis, the authors train layer-specific linear probes on mean-pooled span activations and introduce probe-guided attention masking, which prevents generated tokens from attending to cached positions classified as restricted without modifying the base model.

Experiments across five LLMs on a synthetic Contextual Integrity benchmark demonstrate an average probe accuracy of 90.5%, while 96.2% of 226 context-flip pairs exhibit representation shifts in the expected privacy direction. The proposed method achieves higher Completeness than CI-CoT (32.3% versus 24.7%) with fewer generated tokens (178 versus 475), although at the cost of reduced Utility (50.4% versus 58.1%). The paper further evaluates probe architectures, masking strategies, and computational overhead through ablation studies and runtime measurements.

Critical strengths

S1. Empirical support for linear accessibility of contextual privacy signals

The paper provides meaningful empirical evidence for its central hypothesis rather than relying solely on an assumed linear structure. Linear probes achieve 90.5% average accuracy across five models, while a higher-capacity MLP improves accuracy by only 0.30 percentage points on average in the fixed-layer comparison (Tables 1 and 5). Moreover, removing preceding task context reduces accuracy by 19.4 percentage points, and changing only the task moves representations in the expected privacy direction for 96.2% of context-flip pairs (Tables 2 and 4). These complementary experiments support the claim that contextual privacy information is linearly accessible in the evaluated representations, rather than merely reflecting fixed properties of individual attributes.

S2. Effective inference-time intervention with carefully designed ablations

The proposed method translates the identified privacy signals into a targeted inference-time intervention that suppresses attention to restricted spans without retraining the underlying LLM (Section 5, Eq. 7). Across all five models, attention masking achieves higher Completeness than CI-CoT (32.3% versus 24.7% on average) while using substantially fewer generated tokens (Table 3). Comparisons with alternative probe architectures, soft attention suppression, and oracle masking help distinguish the contributions of representation probing and the masking mechanism (Table 5; Figures 2–3). The inclusion of runtime measurements accounting for probe prefill and masked generation, showing approximately 1.15× overhead relative to the unmasked baseline under the evaluated setting, further strengthens the practical relevance of the contribution (Appendix C, Table 6).

Critical weaknesses

W1. Limited external validity of the structured synthetic evaluation

The main evaluation relies on a synthetic benchmark containing only 73 held-out contexts (398 spans), with information presented as structured attributes (Section 6; Appendix B). The proposed pipeline further isolates each candidate span with its preceding task context, reducing representational interference from unrelated attributes (Section 5). Although this design enables controlled examination of contextual privacy signals, it does not establish whether the approach remains reliable for unstructured documents, ambiguous span boundaries, or disclosures involving relationships among multiple pieces of information. The additional 226 context-flip pairs remain synthetic and similarly structured, leaving the method's applicability to realistic LLM-agent workflows insufficiently validated (Appendix D).

W2. Insufficient validation of privacy labels and potential indirect information leakage

The additional context-flip examples are generated and labeled by GPT-4 Turbo, with structural consistency checks but no reported independent human validation of the contextual privacy judgments (Appendix D). Furthermore, the Integrity evaluation is based on whether restricted attribute values appear in the generated answer, leaving unclear how paraphrased, partial, or inferential disclosures are handled (Section 6). A related limitation arises from the masking mechanism: the prompt is processed without masking its source positions during prefill, so hidden states at later, unmasked positions may already encode information from restricted spans, even when generation-time attention to their original positions is suppressed (Section 5; Appendix B). Consequently, the reported improvements demonstrate suppression of labeled restricted values, but do not establish comprehensive control over indirect information leakage.

W3. Gap between linear accessibility and reliable context-dependent privacy decisions

Although the linear probes achieve 90.5% accuracy on the primary benchmark, their performance on the independently constructed context-flip set is substantially weaker (Table 2). In particular, the reported directional consistency, defined in Eq. 8 as

D=1N∑i=1N1[wℓ⊤Δhi(ℓ)>0],D = \frac{1}{N}\sum _{i=1}^{N}\mathbb{1}\left[\mathbf{w} _\ell^\top \Delta\mathbf{h} _i^{(\ell)} > 0\right],

reaches 96.2%, whereas endpoint classification accuracy is only 74.1% and both contextual judgments are correct for merely 48.5% of pairs. Thus, observing a consistent linear representation shift does not necessarily imply a reliable decision boundary under changing disclosure contexts, especially with thresholds calibrated on the original validation distribution. Moreover, Appendix A derives sufficient conditions for linear separability, such as ηℓ<γℓ\eta _\ell < \gamma _\ell, but these conditions depend on an assumed decomposition and do not independently establish that the privacy-related component can be identified or remains stable across heterogeneous contexts.

W4. Computational efficiency is not sufficiently characterized beyond the evaluated setting

Although the linear classifier itself costs only O(Kd)O(Kd), the method requires separate Transformer prefill computations for candidate spans, with total additional complexity (Appendix C, Eq. 22)

ΔC=O(L∑i=1K(Pid2+Pi2d))+O(LH(N+MN+M2)),\Delta C = O\left(L\sum _{i=1}^{K}\left(P _i d^2 + P _i^2 d\right)\right) + O\left(LH(N + MN + M^2)\right),

where KK is the number of candidate spans, PiP _i is the length of the corresponding isolated probe input, NN is the full prompt length, and MM is the number of generated tokens. The reported average runtime overhead of 1.15× is encouraging, but is measured with only 5.45 candidate spans per context on average and relatively short, batched probe inputs (Table 6). The paper does not characterize how this overhead scales with longer contexts or larger numbers of candidate spans, and its wall-clock comparison is against the unmasked baseline rather than CI-CoT. Consequently, efficiency under the current structured setting is supported, but scalable or end-to-end runtime superiority over reasoning-based alternatives remains insufficiently established.

Questions for authors

Q1. Privacy-label validity and disclosure evaluation (Clarification)

Could the authors clarify whether the newly generated context-flip labels were independently validated against human judgments or an explicit Contextual Integrity annotation protocol? Additionally, how does the Integrity evaluation handle paraphrased, partially disclosed, or inferable private information rather than explicit reproduction of restricted attribute values? This clarification would materially affect how confidently the reported results can be interpreted as privacy protection.

Q2. Reliability of context-dependent decisions (Additional evidence)

Table 2 reports 96.2% directional consistency but only 48.5% accuracy when requiring both contextual classifications to be correct. Could the authors provide a more detailed analysis of false-positive and false-negative errors under context changes, including whether threshold recalibration on held-out contexts substantially improves performance? This would help establish whether the observed linear signal can support dependable disclosure decisions beyond the original benchmark distribution.

Q3. Robustness beyond structured inputs (Additional evidence)

Can the authors provide evidence that probe performance and masking effectiveness remain comparable when applied to unstructured inputs or scenarios requiring joint interpretation of multiple attributes? In particular, evaluating realistic synthetic documents without predefined attribute boundaries would help distinguish genuine contextual privacy reasoning from reliance on the current input structure.

Q4. End-to-end computational cost (Clarification and additional evidence)

Could the authors report a direct end-to-end wall-clock comparison with CI-CoT, including all probe prefill and masking overhead? It would also be valuable to characterize runtime scaling as the number of candidate spans KK and isolated probe-input lengths PiP _i increase. These results would clarify the computational advantage of the proposed method under workloads more representative of practical LLM agents.

3 comments

We thank the reviewer for the careful reading and the constructive comments.

W1/Q3. Limited external validity of the structured synthetic evaluation

We follow the original train, validation, and test splits (590, 66, and 73 task contexts) and the evaluation protocol of [4] The 73 held-out contexts contain 5.45 annotated attributes per context on average, yielding 398 labeled attribute–context pairs for span-level probe evaluation. We measure generation performance (Integrity, Utility, and Completeness) at the context level across the 73 held-out contexts.

The CI benchmark’s training split covers disclosure decisions for individual attributes in structured contexts. The subtler cases raised by the reviewer, such as when several pieces of information jointly reveal a private fact, require training examples that capture those relationships.

Our probe-input template is part of the method. The datasets we considered would require substantial human annotation to fit this template. Annotators would need to verify the extracted attributes and their source positions, check that the target is correctly separated from its context and that the context retains the relevant evidence, and assign a disclosure label to each target in that context.

Our probing and masking approach can serve as a building block for contextual privacy control. Extending it to unstructured documents and harder privacy tasks depends on constructing training examples and labels that capture the relationships that govern disclosure, so that the probe learns the correct direction under our hypotheses and theoretical framework.

W2/Q1.a. Insufficient validation of privacy labels and potential indirect information leakage

For every context-flip pair, human reviewers and GPT-6.1 Sol first independently judged whether the target span was appropriate to disclose in each context, with the original labels hidden from both. The human reviewers then considered the model’s assessments and assigned the final labels. We will describe this validation protocol in the revised manuscript.

W3/Q2. Threshold Recalibration and Error Analysis

We evaluated threshold recalibration on all 226 context-flip pairs (452 endpoints) with the trained probes held fixed. In grouped five-fold evaluation, we kept both contexts of each pair in the same fold and selected one threshold per model using only the calibration folds.

MetricOriginal thresholdsRecalibrated thresholds
Endpoint accuracy74.07%74.42%
Both contexts correct48.50%49.03%
False-negative rate34.07%23.81%
False-positive rate17.79%27.35%

The 0.35 percentage-point gain in endpoint accuracy had a 95% bootstrap confidence interval of [−1.93, +2.13] pp over 2,000 resamples. Leave-one-domain-out evaluation yielded 74.38% accuracy, and a second probe configuration yielded 73.94% before and 73.67% after recalibration.

The main effect was a trade-off between missed restrictions and allowed spans incorrectly flagged as restricted. Targeting a false-negative rate of at most 5% on the calibration folds yielded a held-out false-negative rate of 5.13%, a false-positive rate of 69.03%, and endpoint accuracy of 62.92%. We computed error rates against the original GPT-4 Turbo labels and reported disputed cases separately.

Please see our response to Reviewer 95oh (W2a) for how we interpret the 96.2% directional consistency and what the contextual-shift results support.

W4/Q4.a. Computational efficiency is not sufficiently characterized beyond the evaluated setting

Appendix C reports wall-clock runtimes relative to the unmasked baseline. We now provide a direct end-to-end comparison with CI-CoT based on repeated runs. We report probe preparation separately and include it in the Probe + Mask total. On the evaluated workload, our method achieves a 1.90–2.62× speedup across the five models.

ModelCI-CoT (s)Probe Preparation (s)Probe+Mask Total (s)Speedup
Qwen2.5-7B219.18 ± 0.5015.13112.33 ± 0.301.95×
Qwen2.5-14B385.78 ± 103.4820.34151.10 ± 42.122.55×
Llama-3.1-8B336.47 ± 3.1515.82128.29 ± 1.302.62×
Mistral-7B270.88 ± 0.8814.88142.84 ± 0.491.90×
Gemma-2-9B283.16 ± 0.6617.77136.12 ± 0.552.08×

We will add these measurements to Appendix C.

W2/Q1.b. Potential indirect information leakage

In our template, the attribute list is appended after the task context. Under causal self-attention, the cached states of the preceding task context depend only on tokens up to their own positions. Indirect transfer from a restricted attribute through the prompt cache can therefore occur through later, unmasked positions that attended to it during prefill.

In addition, we allow the model to see and process the spans identified by the probe as part of its input, our intervention is to target the model’s attention to probe-flagged spans when predicting output tokens. As detailed in Appendix C, the mask applies to the final prefill query that predicts the first output token and to every subsequent decoding step. By suppressing direct attention to the flagged spans’ cached keys and values, the method reduces the likelihood that the model draws on restricted information when composing its answer.

W4/Q4.b. Overhead scales with longer contexts or larger numbers of candidate spans

Our task–attribute input template tells the probe exactly which candidate attribute to classify. Each probe input contains the shared task description and a single designated attribute. This keeps each probe pass focused on the information needed to judge that attribute, even when the full generation prompt is much longer. With the shared task description fixed, the additional probe prefill workload depends on the number of candidate attributes and their token lengths.

We report the average time per token for each stage.

ModelProbe Prefill (ms/input token)Total Preparation (ms/input token)Scoring + Mask Build (µs/prompt token)
Qwen2.5-7B0.19470.20060.0383
Qwen2.5-14B0.27560.28160.0503
Llama-3.1-8B0.20460.21020.0400
Mistral-7B0.20260.20780.0391
Gemma-2-9B0.26020.26490.0407
Average0.22750.23300.0417

Across all five models, total preparation takes less than 0.3 ms per input token, and scoring plus mask construction takes only tens of nanoseconds per prompt token.

Thank you again for your valuable comments and constructive suggestions. We have carefully addressed your concerns in our rebuttal and provided additional clarifications where necessary. As the discussion period is limited, we would be most grateful if you could find a moment to look over our response and let us know whether our explanations have adequately addressed your questions. Should any concerns remain, we would be very happy to provide further clarification. Since our experiments are computationally intensive and take considerable time to complete, an early reply would be of great help in allowing us to prepare any additional results you may wish to see within the remaining discussion period. We sincerely appreciate your time and effort in reviewing our paper.

Reviewer_95ohOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether LLMs encode context-dependent privacy judgments at the level of individual information spans. The authors propose the Contextual Linear Privacy Hypothesis, train layer-specific linear probes on span representations, and introduce probe-guided attention masking that prevents generated tokens from directly attending to spans predicted to be restricted. Across five models, the probes achieve roughly 90% classification accuracy. On a separately generated context-flip dataset, probe scores move in the expected direction for 96.2% of pairs, although only 48.5% of pairs are classified correctly under both contexts. The proposed masking method improves average Completeness from 14.5% for the base model and 24.7% for CI-CoT to 32.3%, while generating substantially fewer output tokens than CI-CoT.

Critical strengths
  • This paper studies an interesting problem, moving contextual integrity from a task-level decision to a span-level decision is well motivated. The same piece of information may be appropriate in one task but inappropriate in another, so span-level contextual modeling is a meaningful direction.
  • Probe-guided attention masking is conceptually simple and does not require updating the base model. The distinction from global activation steering is particularly sensible because privacy decisions are inherently selective over individual spans.
  • Results are consistent across several model families, and the runtime analysis is reasonably detailed.
  • This paper investigates context removal, nonlinear probe architectures, soft versus hard attention suppression, activation steering, oracle span selection, and masking coverage. The oracle experiment is particularly informative: oracle masking reaches 43.6% Completeness while largely preserving Utility, suggesting that span-selection errors are a major remaining bottleneck.
Critical weaknesses
  • The main representation claim relies on specially constructed probe inputs. Each attribute is independently paired with the task and passed through the model, rather than probing its representation in the actual full generation prompt. This makes the signal cleaner but weakens the claim that the model naturally “already knows” what not to share during ordinary prompt processing.
  • The contextual-shift result is weaker than the 96.2% number suggests. Although probe scores usually move in the expected direction, endpoint accuracy is only 74.1%, and both contexts are classified correctly for only 48.5% of pairs. A useful missing control is to change the task while keeping the privacy label unchanged.
  • The evaluation is limited to a small synthetic benchmark. The generation test set contains only 73 examples, and the method assumes a highly structured input format. Evaluation on an independent or more realistic privacy benchmark would substantially strengthen the paper. -A simple redaction baseline is missing. Once the probe identifies a restricted span, a natural alternative is to remove or replace that span before generation. This would clarify whether attention masking itself provides meaningful benefits.
Questions for authors
  • What is probe accuracy when representations are extracted directly from the full generation prompt?
  • How does attention masking compare with simply deleting probe-identified restricted spans?
  • The AutoPASTA sentence in Section 2.2 appears grammatically incomplete: “...to improve contextual faithfulness dynamically steer attention toward instructions.”
  • The paper uses α\alpha both for the privacy coefficient and for steering strength in Figure 2a/Table 7, while the main text uses γ\gamma for steering strength. Similarly, the soft-masking bias is denoted β\beta in the text but bb in Figure 2b.
  • Table 5 evaluates Qwen2.5-7B at L21 and Llama-3.1-8B at L25, while Table 1 selects L22 and L17, respectively. The paper does not explain how the Table 5 layers were chosen, which also makes the reported linear-probe numbers differ from Table 1.
6 comments

We thank the reviewer for the careful reading and the constructive comments.

W1/Q1. What is probe accuracy when representations are extracted directly from the full generation prompt?

We extracted representations at the target span positions directly from the full generation prompts and evaluated span-level classification on the same test set. Across five models, mean accuracy was 89.10% with full prompts, compared with 90.45% with isolated inputs. This shows that contextual disclosure labels remain linearly decodable from span representations during full-prompt processing on this benchmark.

ModelIsolated Input Accuracy (%)Full Generation Prompt Accuracy (%)Change (pp)
Qwen2.5-7B89.9586.18−3.77
Qwen2.5-14B91.7189.70−2.01
Llama-3.1-8B89.9588.44−1.51
Mistral-7B89.9591.21+1.26
Gemma-2-9B90.7089.95−0.75
Average90.4589.10−1.36

We pair each target attribute with its task context separately to control irrelevant variation during probe training. In this benchmark’s per-attribute setup, whether a target attribute is appropriate to disclose can be determined from the attribute itself and the task context, including the recipient and purpose. Encoding each pair separately preserves the evidence needed for that decision while removing variation introduced by other attributes and their ordering. As guided by Appendix A, this input construction is designed to make the direction associated with disclosure appropriateness easier for the probe to learn. The comparison shows a modest average benefit from isolation, with the effect varying across models.

W2a. The contextual-shift result is weaker than the 96.2% number suggests.

Our method takes candidate spans with known source positions and the task context needed to assess whether each span is appropriate to disclose. The probe-input template is an explicit step in the method. We place the task context before each candidate span and encode the candidates separately. This design aims to reduce variation from unrelated attributes and their ordering, which can contribute to the projected nuisance term in Appendix A.

Linear probing measures what a restricted predictor can recover from a given representation [1]. Prior work shows that task templates and prompt cues can affect which signals are accessible to a linear readout and how well that readout transfers across input formats [2, 3]. This motivates constructing the probe input explicitly and interpreting our results in light of that input construction.

For raw documents, a trained span extractor or a separate LLM reasoning pass could supply candidate spans, their source positions, and any context needed for judgments involving multiple spans. CI-CoT provides a precedent for explicit per-attribute reasoning before generation. In this extension, the upstream pass would identify and organize the evidence, leaving the disclosure decision to the probe.

Our probing and masking approach can serve as a building block for contextual privacy control. Extending it to cases where disclosure depends on relationships among multiple spans requires training examples that capture those relationships, disclosure labels grounded in the full context, and evaluation of whether the probe learns a privacy direction that tracks disclosure appropriateness in unseen contexts.

W2b. Missing control: change the task while keeping the privacy label unchanged.

Directional consistency measures how often the probe assigns a higher restriction score to the same span in the restricted context than in the allowed context. Correctly classifying both contexts additionally requires the decision threshold to separate the two scores. At the original validation-selected thresholds, mean endpoint accuracy is 74.1%, and both endpoints are classified correctly in 48.5% of pairs.

We constructed the 226 context-flip pairs independently of the original benchmark, using the same input template with newly generated tasks and attributes. We applied the probes and thresholds from the original benchmark directly to these pairs to evaluate transfer. The results show that the expected within-pair score ordering largely transfers to this independently constructed set, while binary decisions at the original thresholds remain less reliable.

Following the suggestion, we added a label-preserving control on the original test set. We used GPT-6.1 Sol to rewrite the task context while keeping the evaluated spans fixed, then manually checked that the original disclosure labels remained appropriate under the rewritten contexts. Mean accuracy changed from 90.45% to 89.95%, a decrease of 0.50 percentage points.

ModelOriginal Test Accuracy (%)Replaced Context Accuracy (%)
Qwen2.5-7B89.9589.45
Qwen2.5-14B91.7191.96
Llama-3.1-8B89.9588.19
Mistral-7B89.9590.45
Gemma-2-9B90.7089.70
Average90.4589.95

W3. The evaluation is limited to a small synthetic benchmark.

We follow the original train, validation, and test splits (590, 66, and 73 task contexts) and the evaluation protocol of [4] The 73 held-out contexts contain 5.45 annotated attributes per context on average, yielding 398 labeled attribute–context pairs for span-level probe evaluation. We measure generation performance (Integrity, Utility, and Completeness) at the context level across the 73 held-out contexts.

Q2. How does attention masking compare with simply deleting probe-identified restricted spans?

We added a deletion baseline using the same probe predictions as our masking method. For deletion, we remove the flagged spans from the prompt before generation.

ModelI (%)U (%)C (%)
Qwen2.5-7B65.7549.3232.88
Qwen2.5-14B64.3843.8430.14
Llama-3.1-8B69.8647.9531.51
Mistral-7B60.2757.5332.88
Gemma-2-9B73.9738.3627.40
Average66.8547.4030.96

Masking achieved higher Utility on all five models, improving mean Utility by 3.0 percentage points. It also achieved higher mean Integrity and Completeness under the existing evaluation.

InterventionIntegrity (%)Utility (%)Completeness (%)
Probe-guided deletion66.8547.4030.96
Probe-guided attention masking67.950.432.3

One possible explanation is that deletion is more sensitive to false positives on task-required spans. If the probe mistakenly flags a required span, deletion removes it before the model processes the prompt. Attention masking keeps the span in the prompt and blocks direct attention to its positions when predicting output tokens, starting with the final prefill query. Earlier prompt computations can therefore retain some task-relevant context from the flagged span in other positions. This mechanism may help explain why masking preserves more required information in our evaluation.

Q3. Grammatically Incomplete

Thank you for catching the incomplete AutoPASTA sentence in Section 2.2. We will correct it in the revision.

Q4. Inconsistent Notation for Steering Strength and Soft-Masking Bias

Thank you for flagging the inconsistent notation. To match the main text, we will use γ for steering strength in Figure 2a and Table 7 and β for the soft-masking bias in Figure 2b.

Q5. Layer Selection

As described in Section 6, we select the probe layer for Table 1 jointly with the regularization strength and decision threshold by maximizing validation accuracy. In Table 5, all three probes use the layer that yielded the highest validation accuracy across MLP-128 and Adapter-r16. We will clarify this rule in Section 6.2.1.

References

[1] Hewitt, J., Ethayarajh, K., Liang, P., and Manning, C. D. (2021). Conditional Probing: Measuring Usable Information Beyond a Baseline. EMNLP. Paper

[2] Zou, A., et al. (2023). Representation Engineering: A Top-Down Approach to AI Transparency. arXiv:2310.01405. Paper, Fig. 13

[3] Ichmoukhamedov, T., and Martens, D. (2025). Exploring the Generalization of LLM Truth Directions on Conversational Formats. arXiv:2505.09807. Paper

[4] Lan, G., et al. (2025). Contextual Integrity in LLMs via Reasoning and Reinforcement Learning. NeurIPS. Paper, §3.1

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.