acceptodds
Withdrawn from ICLR 2027

Privacy Is Elicited, Not Innate: How LLM Agents Form Disclosure Decisions, and Why Leaking Less Is Not Deciding Correctly

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

An agent that acts on personal information fails visibly only after the action, when the information has already leaked. Contextual Integrity (CI) names this failure as a flow of information crossing the norms of its source context. We ask whether an LLM agent represents that norm when it decides to disclose, and where the decision forms. To read the norm and the decision from the same hidden state, we introduce CI-Pairs, variants of PrivacyLens scenarios that each change the flow and keep the rest of the story, so that the norm varies within a scenario. When the model acts, the prompt does not fix the outcome, since in a quarter of PrivacyLens scenarios some samples leak and others withhold, and before the action neither a linear nor a nonlinear probe reads the norm from the residual stream beyond what a text baseline reads from the prompt. When the model is instead asked to judge the same flow, described by its attributes, a linear probe reads the norm before it answers, with an AUROC above a text baseline on the same prompt. The model can thus represent the norm, but acting does not call on it, so its privacy behavior is elicited, not innate. We find that the decision forms inside the reasoning trace, which largely settles it within its first quarter, and that under a privacy instruction, or after fine-tuning with CI theory in the training prompt, a probe reads it from the trace beyond what the text carries. Once the action is written, a text baseline on the same words largely matches the probe, so probe AUROC alone does not show a privacy representation. A privacy instruction lowers leakage on PrivacyLens and on a small out-of-distribution benchmark, whereas fine-tuning lowers it on PrivacyLens alone, in two of our three fine-tuned models, and the one of them whose trace we can probe makes the decision no more readable than the base model does. Under CI, a correct decision also lets through the flows the norm permits, which a benchmark made only of violations cannot test, so we test this on CI-Pairs. There, the interventions that cut violations also cut appropriate disclosures, and none makes the agent detectably better at telling the two apart. Privacy must therefore be elicited, and the interventions that elicit it make the agent disclose less without making it follow the norm more closely, a poor footing for agents that act on personal information.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Agents, Privacy, Contextual Integrity, Activation Probing, Steering Vectors, Information Leakage, Mechanistic Interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_x5VwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

N/A: Please see Weaknesses section.

Critical strengths

N/A: Please see Weaknesses section.

Critical weaknesses

W1: This submission is essentially illegible. The text is extremely dense and badly presented, both in its lexical choices, syntax structure and order of presentation. In the core sections of the paper (3-4), the text becomes even harder to parse with a lot of references to the appendix. Despite the significant amount of time spent of this submission, I was not able to even understand the research gaps this paper is trying to fill, let alone the findings. I am incapable of writing a full review of this paper in its current state. The paper needs a full rewrite and heavy restructuring before it is ready to be reviewed again.

Questions for authors

Q: To what extent was the text of this submission revised by humans given that AI was used for "assisting the interpretation of results, including proposing readings of the data that the authors accepted, modified or rejected" ?

Reviewer_ShU5OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how agents represent privacy norms in their residual stream and reasoning traces, while also studying when it makes a privacy disclosure decision. It also extends PrivacyLens (Shao et al) to also include appropriate disclosure examples to perform studies on model privacy judgment and appropriateness reading while taking into account if agents withhold contextually-appropriate information, and not just if agents reveal less sensitive information. They perform tests over a base unaltered model, and variants that are provided with privacy instructions and are fine-tuned/steered for privacy adherence.

Critical strengths
  • [S1] Addresses a timely problem for contextual privacy decision making: whether models represent and then elicit privacy norms, showing how a privacy decision forms inside the reasoning trace how if a norm is represented during generation, it is not suppressed.
  • [S2] Additional results on how fine-tuning and privacy conscious prompting can lead the model to represent the norm beyond what a text classifier can observe are interesting (although very intuitive and not too surprising, but still a nice illustration), whereas the lack of a privacy instruction yields no better representation of the norm from the residual stream over a text-based baseline. Additional out-of-distribution tests using ConfAIde is also useful.
Critical weaknesses
  • [W1] Severely limited empirical evaluation that is not proven to generalize across architectures: Limited empirical results on an agent based on only one model (Gemma 4 12B) across all experiments, with another model (Qwen 3 14B) used for evaluation for reasoning trace readability analysis, which may not provide a good breadth of results to validate the claimed takeaways and may be model-specific only. I recommend results over 3-4 additional models at least over multiple model families. I understand that given the nature of the study, these models will need to be open source.
  • [W2] Major room for improvement in presentation, potential technical inaccuracies, missing CI pairs description in main text: The writing needs to be careful; currently the prose contains critical technical inaccuracies in the paper.
    • For instance, the authors define the disclosure of any attribute as contributing to the leakage rate (line 117): that is not leakage; it is just disclosure; leakage implies that it is necessarily not intended to be disclosed and therefore leaks. Another potential inaccuracy is in how the authors appear to require withholding a (task-relevant) detail as part of their privacy prompt (lines 174-175): this is incorrect, as in CI, a task in a given context is inherently able to be completed without relying on contextually-inappropriate attributes, and this framing is clear in prior work on privacy-conscious agents ([1, 2]).
    • Some terminology and language need to be better presented and scoped. For instance, the authors say that they include CI theory in the prompt for fine-tuning but it is like saying that they are including an entire field, such as a branch of philosophy or quantum mechanics, in the prompt, as CI is a complex, nuanced area/philosophy of privacy. Instead, they should say that they provide a CI-based instruction within the prompt, which is a more accurate and precise description. The authors need to be careful with their prose; for instance, they say that a probe reads text (when it reads the residual stream) (line 087), which is technically incorrect, even if an informed reader may understand what the authors are trying to convey. Additionally, statements such as "In our fine-tuning runs, lower leakage and a more readable decision come apart" (lines 097-098) are vague and imprecise. Terms like gold action (lines 180-181) are undefined. The paper is presented poorly and could do with a round of in-depth proofreading. These examples are not exhaustive and are only a small fraction of such writing issues. If time permits, I am willing to work with the authors over the rebuttal period to smooth out any such problems in the prose to make it more ICLR-standard.
    • Additionally, and this is important, while CI Pairs is central to their contribution, they do not describe it clearly and prominently in the main text, providing a very brief and vague description instead and almost entirely relegate that to the appendix, which is not acceptable as it is a new key contribution of their work and is used for the main evaluation.
Questions for authors

Can the authors address the chief concerns listed in critical weaknesses and the following additional concerns? W3 and W5 require clarifications and edits to the paper for clarity. W4 may require additional evaluation. I am willing to favorably reassess the paper if my concerns are satisfactorily addressed.

  • [W3] Carefully moderating claims of novelty: In a few instances, the authors risk overclaiming novelty (or at best do not attribute some aspects accidentally). For instance, from lines 127 to 130, they describe action and judgment framings that come from the in-the-wild and probing tests from Shao et al. directly, which they do not acknowledge. In lines 160-164 (describing ConfAIde-Agentic), they contradict themselves, at first saying that they use the PrivacyLens format from Shao et al, and then they call it "our format". It is important to be careful with attribution and such language in an academic submission.
  • [W4] Human validation of judges: Key LLM judges used in the evaluation, such as a disclosure judge (line 199) and a labeling judge (line 152) require human validation (see prior validation of disclosure judges in works like [Das et al., 2025]).
Reviewer_MmfZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors introduce CI-Pairs: a 30 scenario paired resource with 15 single attribute edits, leading to 400+ variants. The study probes Gemma4-12b residual streams for privacy norms and disclosure decisions. The authors show that the privacy norm does not exist when the agent acts but can be read when the model judges a flow, the disclosure decision is generally fixed in the first part of the reasoning trace and prompting and SFt reduce leakage at the cost of appropraite disclosures/utility.

Critical strengths
  • The motivation is clear and contextual privacy is an important area to study
  • The experimental setting is rigorous and concrete especially for prompting. Multiple relevant baselines and trivial settings are reported
  • CI-Pairs is interesting as a resource due to its contrastive nature and matched edits. It could be a good resource for the community
  • Some results are surprising. I was especially interested in the finding that recipient and transmission principle edits do not affect disclousre decisions.
Critical weaknesses
  • The paper is slightly hard to read and follow. Presentation should be improved (the full paper is 60+ pages and quite dense). Main text relies on the appendix too much.
  • I am concerned about the claims made in the paper against the empirical evidence in just one model (Gemma). Qwen replication does not have the same results as Gemma.
  • CI-Pairs essentially only has 30 scenarios and the claims made based on this are very broad/general.
  • Authors should conduct a human-study for appropriate norm flows and leakage/helpfulness, especially given the low statistical confidence in the small study.
  • Finetuning results are reported with 215 scenarios and just a single run, weakening the conclusions.
  • CI-Pairs gap is dominated by information-type edits. The authors should discuss this in more detail.
  • In general, my main concern is the claims and experiments do not align, especially given the small datasets labeled by LLM, underpowered results, single model family.
Questions for authors

See above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.