acceptodds
Under review as a conference paper at ICLR 2027

Sensitivity Awareness in Multi-Turn Interactions: An Information-Theoretic Perspective

26 views

Abstract

Despite their impressive capabilities, large language models (LLMs) remain vulnerable to many adversarial attacks. A key weakness is their lack of sensitivity awareness (SenA), i.e., their inability to consistently enforce access rights when processing and sharing sensitive data in multi-turn interactions. Although this problem has recently gained attention, no unified framework exists for analyzing SenA over multiple turns. We address this by defining hop information leakage, a per-turn measure of information disclosure based on mutual information that composes exactly across turns via the chain rule. Using this notion, we derive an upper bound on an adversary's success probability, directly connecting cumulative information leakage to the attacker's advantage through average conditional min-entropy; empirically demonstrate how an uninformed LLM system can leak sensitive information even if it never states the target value outright; propose BayWatch, an information-theoretic SenA guardrail that reduces the attacker's success rate by up to 45 percentage points compared to the state-of-the-art defense. These contributions offer a principled, empirically vetted basis for understanding the capabilities and fundamental limits of multi-turn adversaries against sensitivity-aware systems.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Sensitivity Awareness, LLM Security, Information Leakage, Mutual Information, Multi-Turn Attacks

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_8kiDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes an information theoretic approach to quantifying per-turn information leakage in LLM interactions, with respect to sensitive attributes such as salary. Based on it, they then establish an upper bound on the success probability of an adversary attempting to access specific private information, demonstrate that naive LLMs leak private information, and propose a framework called BayWatch to "police" LLM outputs in terms of privacy leakage, which they show to substantially improve defense capabilities.

Critical strengths
  • the information theoretic framework for assessing privacy leakage is sound and intuitive (if not hugely theoretically innovative)
  • the topic is an important, underexplored one
Critical weaknesses
  • the GLM 5.2 judge, which is the basis of much of the empirical evaluation, appears to be based on a single greedy decoding run only, in which case it is not possible to distinguish between the model "fluking" the privacy leakage (given that it is over a discrete label space) vs. reliably being able to infer it (over multiple runs) with high probability (properly quantified)
  • the quality of the live judge for SimSUM is never evaluated to establish the validity (or otherwise) of the experimental results
  • many details are missing (e.g. "weak open-source model ... akin to the ones introduced in (Abdelnali et al. 2026)") or unjustified (e.g. why use a zero-shot model rather than fine-tuning on JSON-formatted data for the firewall-based defense?)
  • a more minor fixable thing, but why use accuracy when it is acknowledged in the third paragraph of Section 6 that this isn't the best evaluation metric because a majority-class baseline will produce non-trivial results, and the specific result will depend on the data distribution. perhaps instead use a normalised metric which accounts for the data skew, or some version of relative entropy?
  • obvious results are missing, such as using a LoRA-like method to further improve results; this is acknowledged as a simple extension, in which case why not include it in the paper?

update based on author response: many of the missing details are provided in the updated manuscript, annulling a number of my original concerns re reproducibility/completeness of the manuscript, and strengthening the paper. I have upgraded my score accordingly by one point.

Questions for authors
  • on lines 213-214, what is meant by "even if the user is authorized to access that data"; surely if the user has direct access to the data, there is no need for privacy, or is the point that it users should be protected from inadvertently revealing such information, and it should only be deliberately accessible? seems a minor point to make and more confusing than anything else
  • line 323: "tabular data ... as continuous text" -- this is the entire text serialised? was the original formatting captured in any way?
  • what is the size of the ADI data used for evaluation?
  • do you really only use LLM interactions for three patients as your test set for SimSUM (line 326)? is there an impediment to scaling up this experiment?
  • how is the "simple deterministic SimSUM live judge" implemented? sounds like a rule-based method of some sort, but is there anything evaluation of the quality of the judge?
  • how important a difference is it that there was no training of abstraction-layer networks for JSON-formatted extraction/why did you not do this?
  • why are the results for Kimi K3 worse than the smaller models? is there any analysis of its performance to understand what went on here, as it is a surprising result
  • what is the point of the section on "semantically meaningful coarsening" in Section 6? it seems to say that others have done it, but you can't because of the impoverished label set for the private attributes you focus on, in which case, a footnote rather than the second main paragraph of the Results would seem appropriate; or have I missed something here?
3 comments

We appreciate the reviewer’s comments and feedback, particularly the points raised about our judgment protocols. We will concisely address each related question and weakness one by one and point the reviewer to the relevant portions of our revised manuscript. We have highlighted all changes to the manuscript in blue text. Entirely new and substantially altered tables and figures are indicated by blue-colored captions.

W1 – Fluking the Privacy Leakage. To verify the robustness of our post-hoc judgement protocol, and therefore our results, we annotated all BayWatch-related manuscripts with two additional models: Gemini Argon 4 and Qwen 3.8 Max 0902 (see Section G.7). Overall, the reported metric does not differ by more than 5 percentage points per reported value, while all three judges also show at least substantial agreement on a per-feature basis (see Table 9a and 9b).

W2/Q5 – Live Judge. While a deterministic live judge will negatively impact the attacker’s capabilities, our results show that even such a disadvantaged attacker can challenge our defending system. Still, we agree that readers should be explicitly informed that adversaries could likely have been more successful with an LLM-based live judge. We state this explicitly in our revised manuscript (cf. Section 5.2). Additionally, we describe our live judge procedure in Section G.4 and clarify that we could not thoroughly analyze the quality of the deterministic judge beyond a preliminary assessment of the models’ initial response patterns.

W3 – Unclear Statement. We apologize for the confusion caused by our phrasing. We rephrased the corresponding paragraph (starting at line 355) and hope this makes it easier to parse.

W3/W5/Q6 – Firewalls and LoRA Extension. As outlined in the original Firewalls paper, developing domain-specific layers requires paired interaction samples for developing the abstraction rules. Since ADI was used only for single-turn interactions, and SimSUM was never used the way we do, creating such a large demonstration set is infeasible. Furthermore, due to the structured design of our data, we have enough information to define adequate input abstraction rules – we only forward portions of the request to our system that actually refer to the available features. Finally, even a Firewall variant that perfectly abstracts input messages (beyond our deterministic scheme, if even possible) while perfectly coarsening retrieved sensitive data still faces the same issue: the mechanism does not introduce any awareness of leakages caused by highly correlated, non-sensitive data points. As such, we deem this zero-shot variant to sufficiently demonstrate the core strengths (very low direct leakage) and weaknesses (high partial leakage) of LLM firewalls.

The lack of demonstration data also made it difficult to test the effects of LoRA-based optimization. However, our new simulation-based analysis, which simulates interactions between an idealized attacker and defender, provides much more insight into how well a highly optimized BayWatch defender can protect sensitive data (see Sec. 6.1, “Can BayWatch Really Handle a Belief-Guided Adversary?” of the revised manuscript).

W4 – Choice of Metric. To account for the prior baseline, we now also report the metric ``lift'', describing the relative increase over the guessing baseline, throughout our tables, most notably when discussing our simulation-based results.

Q1 – Authorized Access. We apologize for the confusion caused by our phrasing. However, it is important to note that sensitive data can be leaked through the accumulation of authorized requests for non-sensitive information. Using ADI as an example: aside from a single feature ({salary, department, marital status}), our adversary can legally retrieve all other features. Yet, without accounting for the downstream effects (i.e., partial leakage) of sharing such data, an adversary could still precisely deduce the target’s sensitive attribute.

Q2 – Unstructured Text Data of SimSUM. We added a sample patient summary to the Appendix (Section F). To summarize, we left each existing entry of a patient’s report untouched, and added any new information as a dedicated (short) paragraph to the existing note.

Q3 – Size of ADI. For each target feature, we attacked four unique ADI employees, totaling 12 targets across three features (5 runs per individual). The exact features of our ADI targets are summarized in Table 2 of our Appendix.

Q4 – Number of SimSUM patients. We use four SimSUM patients (see Table 3), with each patient representing a unique profile (pneumonia/common cold patient, smoker with COPD, healthy patient). This number of targets represents the number of employees we target per ADI feature. Since our adversary essentially only tries to reconstruct a single, grouped target feature, we deemed this sufficient to support our claims from our ADI experiments.

Q7 – Kimi K3. It is important to differentiate between complex defense strategies that use a smaller model (BayWatch and Firewalls Lite) and baselines of smaller models (marked as GPT-OSS 20B and 120B), in which the model is provided only with the access-rights rules. The former category outperforms Kimi K3 despite relying on a smaller model, because both strategies incorporate mechanisms that increase security without requiring complex reasoning capabilities (i.e., the firewall layers and the BN-based modeling of partial leakage). Meanwhile, Kimi K3 performs as well as or better than the other two baselines in 3/4 settings, and only performs slightly worse in the ADI salary experiments in the worst case.

Q8 – Importance of Coarsening. Because LLM firewalls represent a strong, state-of-the-art solution for protecting sensitive data in agent-to-agent communication, we felt it was necessary to highlight the downside of relying on semantic coarsening (a core concept in the data abstraction layer) when the to-be-protected feature has very few possible values. While this may seem obvious, we viewed it as important enough to dedicate a smaller paragraph to this simple but crucial insight.

Thank you for the detailed response and revised manuscript. I have updated my score based on this extra information, and flagged this in the body of the review.

Reviewer_wsBkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates sensitivity awareness in multi-turn LLM interactions, focusing on how adversaries can infer protected information through accumulated disclosures of correlated attributes. The authors introduce an information-theoretic framework that quantifies per-turn leakage using conditional mutual information and relates cumulative leakage to adversarial inference success. They further propose BayWatch, a defense that uses Bayesian networks and pointwise leakage monitoring to prevent sensitive attribute inference. Experiments on corporate and medical data benchmarks suggest that BayWatch reduces direct and indirect information leakage compared with the evaluated baselines.

Critical strengths

S1. An important problem beyond direct sensitive-information disclosure.

The paper addresses a meaningful privacy risk in LLM-based systems: adversaries may infer protected information from multiple individually permissible disclosures, even when the sensitive attribute is never explicitly revealed. Extending sensitivity awareness to adaptive multi-turn interactions is well motivated and relevant to practical applications involving private data.

S2. A principled information-theoretic formulation.

The authors model adversaries through belief updates and use conditional mutual information to quantify information leakage across interactions. The connection between cumulative leakage and adversarial inference success provides a clear mathematical perspective on the problem.

S3. A concrete defense framework with encouraging empirical results.

BayWatch combines response inspection with Bayesian-network-based inference and pointwise leakage monitoring. The experiments distinguish direct disclosure from inference through correlated attributes and include an analysis of the privacy–utility trade-off. These results demonstrate the potential benefits of explicitly monitoring accumulated information disclosure.

Critical weaknesses

W1. A substantial gap between the information-theoretic adversary model and the evaluated attack strategies.

The paper formalizes an adaptive adversary that maintains beliefs about protected attributes and progressively reduces uncertainty through multi-turn interactions. However, the empirical evaluation primarily relies on existing multi-turn jailbreak methods, including Crescendo, TAP, and generic red-teaming attacks. These methods are not explicitly designed to maximize information gain or optimize the posterior probability of protected attributes.

The authors also acknowledge that the original attacks rarely attempted to collect correlated information, motivating the addition of Partial Crescendo for SimSUM. This suggests that the evaluated attacks may not adequately represent the adversary motivated by the theoretical framework.

A belief-guided adaptive attacker that selects queries based on expected information gain or posterior confidence would provide a more direct and potentially stronger evaluation. Without such an attacker, it remains unclear whether BayWatch is robust against adversaries that explicitly exploit the information-theoretic vulnerabilities identified by the paper.

W2. The evaluation does not sufficiently isolate the distinctive contribution of adaptive multi-turn interactions.

Although the paper emphasizes information accumulation across multiple turns, some protected attributes can already be inferred from a single highly correlated feature. For example, revealing an employee's executive role may almost completely determine their salary range under the constructed Bayesian network.

The current experiments do not systematically compare single-turn inference, non-adaptive multi-turn information collection, and adaptive multi-turn attacks under matched information-access budgets.

Consequently, it is difficult to determine how much additional privacy risk arises specifically from adaptive multi-turn interaction, rather than from correlations that can already be exploited in a single exchange. Such controlled comparisons would better establish the significance of the proposed multi-turn formulation.

W3. Strong dependence on accurate Bayesian networks limits practical applicability and weakens the evaluation.

BayWatch relies on a Bayesian network that accurately captures the conditional relationships between protected and unprotected attributes. The experiments use synthetic or semi-synthetic environments with known probabilistic dependencies, and the evaluation deliberately selects target profiles whose disclosed attributes are strongly correlated with protected information.

Furthermore, the defense and the partial-leakage evaluation rely on the same underlying ground-truth probabilistic structure. While this is useful for controlled experiments, it leaves open whether the reported effectiveness would generalize when the defender's estimated Bayesian network differs from the actual data distribution.

The paper lacks sufficient experiments involving noisy, incomplete, or misspecified Bayesian networks. Such robustness evaluations are essential for assessing whether the proposed approach can provide reliable protection in realistic deployments.

Questions for authors

Q1. Why does the evaluation primarily rely on conventional multi-turn jailbreak attacks instead of belief-guided adaptive adversaries?

Given that the theoretical framework explicitly models adversarial belief updates, have the authors considered attacks that select queries by maximizing expected information gain or posterior confidence in protected attributes? How would BayWatch perform against such attacks?

Q2. How much additional leakage is caused specifically by adaptive multi-turn interactions?

Could the authors compare single-turn inference, non-adaptive multi-turn attribute collection, and adaptive multi-turn attacks under matched query or information-access budgets? This would help distinguish cumulative multi-turn vulnerabilities from single-turn inference using highly correlated attributes.

2 comments

We thank the reviewer for the detailed feedback and questions, both of which we address individually. We have highlighted all changes to the manuscript in blue text. Entirely new and substantially altered tables and figures are indicated by blue-colored captions.

W1/Q1 – Belief-Guided Adversaries. While we believe our existing empirical results (and our original residual leakage analysis) show BayWatch’s ability to protect against belief-guided adversaries, we agree this concern warrants further investigation. Hence, we investigate this question explicitly with a simulation-based analysis of how BayWatch’s leakage rate changes when facing belief-guided adversaries (see Sec. 6.1, “Can BayWatch Really Handle a Belief-Guided Adversary?” of the revised manuscript). Essentially, we compared how the feature release behavior of a BayWatch-based defending system would differ when facing an adversary that requests features (i)(i) in a random order, (ii)(ii) maximizes mutual information, and (iii)(iii) maximizes the confidence in its top guess. Overall, we observe that belief-guided adversaries do not substantially outperform our random baseline, indicating that BayWatch is well-equipped to handle such attackers.

Note: Because this analysis provides more insight into this issue than our original residual leakage assessment, we removed the corresponding paragraph and replaced the original Fig. 3b with a figure summarizing our new belief-guided results.

W2/Q2 – The Contribution of Multi-Turn Interactions. It is true that the role attribute alone is sufficient to deduce an individual's department and that partial leakage (even in the other two ADI features) does not substantially move throughout our simulation-based analysis. However, this does not mean that multi-turn interaction does not introduce any additional risks. To illustrate our point, we also added Table 12 (in the Appendix) showing the fraction of i-long interactions of our “No Gate” defender that would cross our A-PML tau-threshold of 0.5, i.e., how many produced outputs at step i would put the sensitive data at risk of being inferred by an adversary. These results provide a clearer picture. While department remains vulnerable to single-turn attacks (with both MI and Conf. consistently exceeding the τ\tau-threshold when targeting this feature), extending the interaction from one to two steps substantially increases the proportion of risky outputs for other features. This effect is particularly pronounced for marital status, where the proportion of outputs at risk more than triples after just one additional interaction step. For salary, the proportion of risky outputs continues to increase through the fourth interaction step. This observed exposure is cumulative and is completely negated by BayWatch's A-PML-based mechanism.

W3 – Dependence on Accurate Network. To evaluate BayWatch in scenarios where the defender lacks access to the ground-truth network, we simulate interactions between our Conf. adversary and a BayWatch defender who can only access a noisy network (see Sec. 6.1, “Do Modeling Errors Negatively Affect BayWatch?”, in the revised manuscript). Overall, we find that BayWatch’s leakage rate remains substantially stable despite modeling inaccuracies. However, noisy modeling can negatively affect the lifeguard’s utility (higher refusal rate) and weaken the guard’s ability to accurately enforce its A-PML-based threshold.

If we have addressed your concerns, we politely and respectfully request a reassessment of the rating in light of our response. Thank you for your time and consideration.

Reviewer_piYNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies sensitivity awareness in multi-turn interactions, and finds that even when the model never directly states the target value, adversaries can still infer it step by step from correlated features, and that existing defenses handle partial leakage much worse than direct disclosure. The paper proposes BayWatch, in which a lifeguard intercepts the victim's outputs before release and lets them through according to a risk threshold; defines the hop information leakage and Accounted Pointwise Maximal Leakage (A-PML); proves upper bounds on the attack success probability and the SenA advantage (Proposition 1–2, based on average conditional min-entropy and the M-ary Fano function); and conducts attack-defense experiments on the ADI and SimSUM benchmarks, demonstrating that BayWatch outperforms Firewalls Lite and a pure-LLM baseline on the worst case of partial leakage, though it is not uniformly best on direct disclosure.

Critical strengths
  1. The theoretical framework is clear, and the relationship between assumptions and conclusions is explicit: Assumption 1 (queries cannot directly depend on the secret) states the premise for the additivity of expected leakage; Proposition 3 is stated separately from the main-text Proposition 1 (Fano upper bound), giving a clear logical structure.
  2. Direct leakage and partial leakage are measured separately: The experimental design distinguishes the two types of leakage, finding that without defense, partial leakage is significantly higher than direct leakage (e.g., for Kimi K3 on ADI department, Vanilla direct is 0.05 vs. partial 0.85), highlighting the inadequacy of only preventing the secret from being stated directly.
  3. BayWatch's design motivation is sound and includes several practical considerations: The accounted prior mechanism of A-PML avoids over-blocking rare values; τ and α provide a tunable safety-utility trade-off; the lifeguard's clarification requests are not written into the victim's history, preventing exploitation by a compromised victim.
Critical weaknesses
  1. The "up to 45 percentage points reduction" in the Abstract lacks corresponding evidence: Neither Section 6 nor Table 4–5 identifies which cell corresponds to these 45 points. In Table 4, the ADI salary W/Partial, Crescendo cell is 0.75 vs BayWatch's 0.25 (50 points), and in Table 5 it is 0.6 vs 0.3 (30 points); neither matches 45.
  2. The theoretical propositions are not directly tested in the experiments: The metrics in Section 5–6 rely on a post-hoc judge (GLM 5.2) extracting evidence that is then evaluated by the ground-truth network; the hop leakage of conversations or numerical comparisons between the Proposition 1–2 upper bounds and measured success rates are not computed. There is a lack of quantitative bridging between theory and experiment.
  3. The defender relies on a ground-truth Bayesian network, and the deployment limitation is not stated: Section 5.1 explains that dataset selection depends on whether CPDs can be obtained, and ADI's network is a semi-approximated network rather than the original distribution. The conclusion does not explicitly state "the defender must hold an accurate joint probability distribution" as a deployment prerequisite.
Questions for authors
  1. Which specific set of experimental data in the Abstract corresponds to the "up to 45 percentage points reduction"? The largest differences in Table 4–5 are 50 and 30 percentage points respectively; how is 45 computed?
  2. Can BayWatch be evaluated without relying on a ground-truth Bayesian network? In real deployments defenders often cannot obtain accurate CPDs; how would BayWatch's performance degrade in that case?
  3. The attacks in the experiments mainly optimize direct leakage; why not design proactive attacks targeting partial leakage? Section 6 admits the attacks mainly optimize direct leakage, with only a partial crescendo added on SimSUM. Is there a corresponding adjacent-information-gathering attack on ADI? If so, would BayWatch's advantage still hold?
2 comments

We thank the reviewer for their positive outlook on our work and for helpful feedback that allowed us to improve it further. We will address the weaknesses and raised questions one by one. We have highlighted all changes to the manuscript in blue text. Entirely new and substantially altered tables and figures are indicated by blue-colored captions.

W1/Q1 – The 45% Gap. We apologize for the confusion caused by not explicitly outlining the origin of our key performance number. The 45% gap represents the difference between the worst performance of BayWatch in ADI salary W/Partial (block rate 0.7 against vanilla) and the worst performance of Firewalls Lite in the same category (0.25 against crescendo). The revised manuscript now mentions the origin of the exact gap in the introduction of the results section (lines 420-422).

W2 – Quantitative Bridging between Theory and Experiments. To showcase how our theory can be applied in empirical settings, we added a section to our Appendix (Section G.6) that describes how to compute the adversary’s likelihood of inferring the secret information (Proposition 1). Furthermore, we briefly argue that computing this likelihood actually has a meaningful impact on data holders: it allows them to identify settings where a gate against such belief-guided adversaries is necessary.

W3/Q2 – How does BayWatch Perform When Facing Suboptimal Modeling? To investigate the reliance on accurate Bayesian network representation, we replaced our simulation-based analysis of the defense’s residual leakage with a set of experiments that assess how noise-based modeling errors impact a BayWatch defender (see Sec. 6.1 “Do Modeling Errors Negatively Affect BayWatch?” of the revised manuscript). To summarize: while the BayWatch leakage rate remains substantially stable, noisy modeling can negatively affect the lifeguard’s utility (higher refusal rate) and weaken the guard’s ability to accurately enforce its A-PML-based threshold.

Q3 – Adjacent Information Gathering. While our residual leakage results (the original Fig. 3b) hint at BayWatch’s ability to defend against a perfect, belief-guided adversary, we agree that this initial finding does not adequately answer the question. As such, we replaced our residual leakage analysis with a similar, simulation-based analysis to explore this question explicitly (see Sec. 6.1, “Can BayWatch Really Handle a Belief-Guided Adversary?” of the revised manuscript). To summarize, our data show that the leakage rate of our defender does not substantially differ between adaptive, belief-guided adversaries and random adversaries. The new Fig. 3b represents an overview of the ASR of a simulation-based, belief-guided adversary against a defender with and without a partial-leakage-aware gate.

We appreciate your positive evaluation of our work. As the rebuttal phase ends soon, we wanted to ask whether there are any points we can clarify and, if we have addressed your concerns, kindly ask you to reassess your scores.

Comments on the paper

1 comment

Dear reviewers, as the deadline is approaching, please check the rebuttal and consider participating in the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.