- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces MuPPET, a benchmark of 562 synthetic workplace conversations for evaluating contextual privacy when an assistant speaks on behalf of one user to multiple recipients. It compares disclosure in one-to-one and group settings, evaluates eight models and two prompting defenses, and uses diagnostic probes to analyze failures in participant tracking and contextual privacy reasoning.
Critical strengths
- The paper examines a setting in which an assistant with access to one user’s private context communicates with a group. In this setting, the same response may be appropriate for some recipients but inappropriate for others. Explicitly separating the owner’s private memories from the shared conversational context makes this concern concrete.
- The one-to-one versus multi-party comparison keeps the conversation and memories fixed and changes only the intended audience, which provides a clean control. The increases for several models, including a change in model ranking are informative.
- The paper provides detailed generation and evaluation prompts, measures utility alongside privacy, and validates both judges against human annotations. This makes the pipeline inspectable.
Critical weaknesses
- The leakage label records disclosure, not a validated norm violation. Each seed's sensitive attribute is simply declared "not intended for disclosure" (L1605), and any response that reveals it counts as a leak (L212–214). The judge sees only the topic, the memories and the response (App. E.1), not the conversation or who already knows. The main-setting prompt also tells the model to "use these memories to personalize responses" and gives no privacy guidance (App. G). The human validation (App. F.1) only checks whether the attribute was revealed, not whether revealing it was inappropriate. Every reported number depends on this label, so the claim to test "implicit social norms" (Table 1, L149–150) is unsupported until humans validate the appropriateness labels.
- The central claim that multi-party settings are harder because of party tracking is not supported. Table 2 adds an explicit note about what not to share, worded differently in the two arms (L264–269). The headline 1.9× increase is therefore not measured in the benchmark's implicit setting, where leakage is much higher (GPT-5.5: 27.4% in Table 2 vs. 49.0% in Table 3). The test uses only 5 runs, so every "significant" raw p-value is 0.03125, the smallest possible, which becomes exactly 0.05 after BH correction (App. H). Three of eight models leak less in the multi-party setting. The Sec. 5.4 probes have no prompts, ground truth or scoring criteria, and about three-quarters of GPT-5.5's leaks fall in the unexplained residual category (Fig. 3). So neither the multi-party effect nor its party-tracking explanation is established, and these are the paper's main case for a separate benchmark.
Questions for authors
-
Why should a workplace coordination assistant access and retain a coworker's health, immigration or pregnancy information, rather than only the resulting work constraint?
-
How is "inappropriate disclosure" defined, and on what basis? If the non-disclosure labels (L1605) are stipulated in advance, how did scenario construction ensure that disclosure is actually inappropriate in each generated context, e.g., when several group members already know the attribute (App. B)? If appropriateness depends on context, what rules let the judge tell inappropriate from legitimate disclosure, given that it sees no recipient or context information (App. E.1) and F.1 only checks whether the attribute was revealed?
-
Why was Gemini 2.5 Pro evaluated instead of the already available Gemini 3 Pro / 3.1 Pro? Does the headline claim, which relies on Gemini 2.5 Pro's ×1.9 increase (L301–308), still hold for Gemini 3.x?
-
What are the exact prompts, ground-truth answers and scoring rules for each of the four probes (L420–427)? Without them, Figure 3 cannot be reproduced.