- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes an information theoretic approach to quantifying per-turn information leakage in LLM interactions, with respect to sensitive attributes such as salary. Based on it, they then establish an upper bound on the success probability of an adversary attempting to access specific private information, demonstrate that naive LLMs leak private information, and propose a framework called BayWatch to "police" LLM outputs in terms of privacy leakage, which they show to substantially improve defense capabilities.
Critical strengths
- the information theoretic framework for assessing privacy leakage is sound and intuitive (if not hugely theoretically innovative)
- the topic is an important, underexplored one
Critical weaknesses
- the GLM 5.2 judge, which is the basis of much of the empirical evaluation, appears to be based on a single greedy decoding run only, in which case it is not possible to distinguish between the model "fluking" the privacy leakage (given that it is over a discrete label space) vs. reliably being able to infer it (over multiple runs) with high probability (properly quantified)
- the quality of the live judge for SimSUM is never evaluated to establish the validity (or otherwise) of the experimental results
- many details are missing (e.g. "weak open-source model ... akin to the ones introduced in (Abdelnali et al. 2026)") or unjustified (e.g. why use a zero-shot model rather than fine-tuning on JSON-formatted data for the firewall-based defense?)
- a more minor fixable thing, but why use accuracy when it is acknowledged in the third paragraph of Section 6 that this isn't the best evaluation metric because a majority-class baseline will produce non-trivial results, and the specific result will depend on the data distribution. perhaps instead use a normalised metric which accounts for the data skew, or some version of relative entropy?
- obvious results are missing, such as using a LoRA-like method to further improve results; this is acknowledged as a simple extension, in which case why not include it in the paper?
update based on author response: many of the missing details are provided in the updated manuscript, annulling a number of my original concerns re reproducibility/completeness of the manuscript, and strengthening the paper. I have upgraded my score accordingly by one point.
Questions for authors
- on lines 213-214, what is meant by "even if the user is authorized to access that data"; surely if the user has direct access to the data, there is no need for privacy, or is the point that it users should be protected from inadvertently revealing such information, and it should only be deliberately accessible? seems a minor point to make and more confusing than anything else
- line 323: "tabular data ... as continuous text" -- this is the entire text serialised? was the original formatting captured in any way?
- what is the size of the ADI data used for evaluation?
- do you really only use LLM interactions for three patients as your test set for SimSUM (line 326)? is there an impediment to scaling up this experiment?
- how is the "simple deterministic SimSUM live judge" implemented? sounds like a rule-based method of some sort, but is there anything evaluation of the quality of the judge?
- how important a difference is it that there was no training of abstraction-layer networks for JSON-formatted extraction/why did you not do this?
- why are the results for Kimi K3 worse than the smaller models? is there any analysis of its performance to understand what went on here, as it is a surprising result
- what is the point of the section on "semantically meaningful coarsening" in Section 6? it seems to say that others have done it, but you can't because of the impoverished label set for the private attributes you focus on, in which case, a footnote rather than the second main paragraph of the Results would seem appropriate; or have I missed something here?
3 comments
Q7 – Kimi K3. It is important to differentiate between complex defense strategies that use a smaller model (BayWatch and Firewalls Lite) and baselines of smaller models (marked as GPT-OSS 20B and 120B), in which the model is provided only with the access-rights rules. The former category outperforms Kimi K3 despite relying on a smaller model, because both strategies incorporate mechanisms that increase security without requiring complex reasoning capabilities (i.e., the firewall layers and the BN-based modeling of partial leakage). Meanwhile, Kimi K3 performs as well as or better than the other two baselines in 3/4 settings, and only performs slightly worse in the ADI salary experiments in the worst case.
Q8 – Importance of Coarsening. Because LLM firewalls represent a strong, state-of-the-art solution for protecting sensitive data in agent-to-agent communication, we felt it was necessary to highlight the downside of relying on semantic coarsening (a core concept in the data abstraction layer) when the to-be-protected feature has very few possible values. While this may seem obvious, we viewed it as important enough to dedicate a smaller paragraph to this simple but crucial insight.
Thank you for the detailed response and revised manuscript. I have updated my score based on this extra information, and flagged this in the body of the review.
We appreciate the reviewer’s comments and feedback, particularly the points raised about our judgment protocols. We will concisely address each related question and weakness one by one and point the reviewer to the relevant portions of our revised manuscript. We have highlighted all changes to the manuscript in blue text. Entirely new and substantially altered tables and figures are indicated by blue-colored captions.
W1 – Fluking the Privacy Leakage. To verify the robustness of our post-hoc judgement protocol, and therefore our results, we annotated all BayWatch-related manuscripts with two additional models: Gemini Argon 4 and Qwen 3.8 Max 0902 (see Section G.7). Overall, the reported metric does not differ by more than 5 percentage points per reported value, while all three judges also show at least substantial agreement on a per-feature basis (see Table 9a and 9b).
W2/Q5 – Live Judge. While a deterministic live judge will negatively impact the attacker’s capabilities, our results show that even such a disadvantaged attacker can challenge our defending system. Still, we agree that readers should be explicitly informed that adversaries could likely have been more successful with an LLM-based live judge. We state this explicitly in our revised manuscript (cf. Section 5.2). Additionally, we describe our live judge procedure in Section G.4 and clarify that we could not thoroughly analyze the quality of the deterministic judge beyond a preliminary assessment of the models’ initial response patterns.
W3 – Unclear Statement. We apologize for the confusion caused by our phrasing. We rephrased the corresponding paragraph (starting at line 355) and hope this makes it easier to parse.
W3/W5/Q6 – Firewalls and LoRA Extension. As outlined in the original Firewalls paper, developing domain-specific layers requires paired interaction samples for developing the abstraction rules. Since ADI was used only for single-turn interactions, and SimSUM was never used the way we do, creating such a large demonstration set is infeasible. Furthermore, due to the structured design of our data, we have enough information to define adequate input abstraction rules – we only forward portions of the request to our system that actually refer to the available features. Finally, even a Firewall variant that perfectly abstracts input messages (beyond our deterministic scheme, if even possible) while perfectly coarsening retrieved sensitive data still faces the same issue: the mechanism does not introduce any awareness of leakages caused by highly correlated, non-sensitive data points. As such, we deem this zero-shot variant to sufficiently demonstrate the core strengths (very low direct leakage) and weaknesses (high partial leakage) of LLM firewalls.
The lack of demonstration data also made it difficult to test the effects of LoRA-based optimization. However, our new simulation-based analysis, which simulates interactions between an idealized attacker and defender, provides much more insight into how well a highly optimized BayWatch defender can protect sensitive data (see Sec. 6.1, “Can BayWatch Really Handle a Belief-Guided Adversary?” of the revised manuscript).
W4 – Choice of Metric. To account for the prior baseline, we now also report the metric ``lift'', describing the relative increase over the guessing baseline, throughout our tables, most notably when discussing our simulation-based results.
Q1 – Authorized Access. We apologize for the confusion caused by our phrasing. However, it is important to note that sensitive data can be leaked through the accumulation of authorized requests for non-sensitive information. Using ADI as an example: aside from a single feature ({salary, department, marital status}), our adversary can legally retrieve all other features. Yet, without accounting for the downstream effects (i.e., partial leakage) of sharing such data, an adversary could still precisely deduce the target’s sensitive attribute.
Q2 – Unstructured Text Data of SimSUM. We added a sample patient summary to the Appendix (Section F). To summarize, we left each existing entry of a patient’s report untouched, and added any new information as a dedicated (short) paragraph to the existing note.
Q3 – Size of ADI. For each target feature, we attacked four unique ADI employees, totaling 12 targets across three features (5 runs per individual). The exact features of our ADI targets are summarized in Table 2 of our Appendix.
Q4 – Number of SimSUM patients. We use four SimSUM patients (see Table 3), with each patient representing a unique profile (pneumonia/common cold patient, smoker with COPD, healthy patient). This number of targets represents the number of employees we target per ADI feature. Since our adversary essentially only tries to reconstruct a single, grouped target feature, we deemed this sufficient to support our claims from our ADI experiments.