- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies what the authors call need-to-know sanitization. A model receives business records plus a sharing context, meaning a recipient and a purpose. It must rewrite the records so that facts the purpose needs are kept and facts the recipient should not see are removed. The main point is that the same fact can be required for one recipient and forbidden for another.
To study this, the authors build PRIVASIS ENTERPRISE, a synthetic corpus of business records. Each case starts from a fictional event and a fact ledger. The ledger also stores derived facts, each computed from two base facts, so indirect leaks can be detected. Two or three records are written from different employees’ points of view. The same records are then paired with a higher-need and a lower-need context, and an LLM labels each fact as PROTECT, RETAIN or OPTIONAL. The release has 9,000 training contexts with verified reference rewrites and 294 held-out evaluation contexts.
Critical strengths
The problem formulation is the strongest part of the paper. Earlier sanitization benchmarks such as TAB, NAP² and Privasis fix what must be removed in advance. Contextual integrity benchmarks such as ConfAIde, PrivacyLens and CI-Work vary the context, but they do not ask a model to rewrite one fixed set of records in two opposite ways. The strict-swap pairing makes this testable. It separates a model that understands the recipient from one that just removes “sensitive-looking” content. The SWAP metric follows directly from this design and is easy to interpret. The fact ledger with derived facts is also a good idea. It lets the benchmark catch leaks that happen through arithmetic or logical combination, which span-level annotation cannot capture.
Critical weaknesses
The ground truth is an LLM’s normative judgment. Every PROTECT/RETAIN/OPTIONAL label comes from one gpt-oss-120b call per context, followed by deterministic fixes. In the context-only track, the model is asked to recover these labels without seeing them. A low score can therefore mean that the model disagrees with gpt-oss-120b, not that it fails to reason about need-to-know. Many of these decisions are not clear-cut. In Figure 4, it is not obvious to me that approved overtime hours should be hidden from an insurance claims manager. One could argue that it matters for coverage. The human validation is meant to address this, but it is reported in a way that is hard to assess.
Questions for authors
Please report inter-annotator agreement before discussion for the boundary review. Who were the annotators, and did they see the constructed labels during the review? How many labels did they disagree on initially, and how were these resolved?