acceptodds
Under review as a conference paper at ICLR 2027

Challenges in Evaluating Contextual Integrity for Large Language Models

Abstract

Modern large language models are powerful tools that can benefit consumers across many domains and applications, but their ability to interact with and use personal information creates a risk of privacy violations when information is improperly shared. This has led to the development of privacy benchmarks for LLMs based on contextual integrity, a framework for evaluating privacy in terms of information flow within specific contexts. We identify key limitations of these benchmarks, such as missing data/code, reproducibility, and incomparable results, that make it challenging to understand how evaluative claims for contextual integrity are interpreted. We further present a critical analysis of three open-source benchmarks with provided data, focusing on areas like data quality, variance, and metric choice. These findings provide additional context for existing benchmarks, informing both practitioners who use them and future benchmark developers about opportunities to improve how benchmarks can more accurately assess contextual integrity in LLMs.

Primary area:
datasets and benchmarks
Keywords:
contextual integrity, large language models, benchmark evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9dE5OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a meta-evaluation of three benchmarks grounded in contextual integrity (CI) with six open-weight models. The analysis reveals five challenges, including data quality, variance, metric choice, CI judgement, and reproducibility. It concludes with a checklist for benchmark designers and users.

Critical strengths
  1. This is a timely work. CI benchmarks are increasingly cited in the studies of agent privacy behavior, so it is necessary to scrutinize the measurement validity.
  2. Identifying and quantifying the inconsistency and limited diversity in synthetic data is a useful contribution.
  3. Calling for the attention to statistical rigor is a necessary and helpful practice.
Critical weaknesses

1. Data quality findings lack an impact analysis

The paper documents inconsistency and limited diversity in the synthetic data, yet an important missing piece is further tests on the impact of such data quality issue, e.g., whether fixing these issues changes the conclusions in the original benchmark paper.

Specifically, Fig. 1 shows one profile (Troy Salazar) accounts for 36.62%36.62\% misaligned attributes. So a straightforward ablation would be removing the inconsistent profiles (or at least the most affected profile) and recomputing privacy violation rates in the benchmarks. Moreover, constructing a more diverse scenario set would test whether conclusions are stable under greater diversity.

2. Benchmark-specific issues are conflated with general LLM-as-judge limitations

Some identified problems are inherent to any use of LLMs for labeling and judging, while others stem from specific benchmark design choices. The current analysis does not separate them, so readers cannot tell where the problems actually originate.

3. Several risks are asserted rather than demonstrated.

(a) labeler nondeterminism is raised as a concern, yet the authors ran 10 labeling trials and could directly report label stability;

(b) the claim that discarded tuples represent “more challenging and realistic” cases is not supported beyond the observation that violation rates rise under relaxed labeling;

(c) the interpretation of unshared necessary attributes as retrieval failure (Sec. 3.4) is not isolated from alternative explanations such as deliberate omission, over-inclusive labels, or judge false negatives;

(d) run-to-run variability is discussed (Sec. 3.5) but not measured in the authors’ own experiments.

4. The metric comparison partly reflects definitional differences

The headline range for Gemma4-31B spans both average vs. worst-case aggregation and strict vs. relaxed labeling. Worst-case-over-trials metrics will exceed averages by construction. Relaxed labeling adds ambiguous tuples and changes the denominator. That these numbers differ is expected.

The more meaningful question is whether metrics intended to measure the same construct disagree on rankings. Appendix F suggests the violation rankings are in fact quite stable (Gemma4-31B > Gemma4-26B > Llama4 under nearly every metric). The paper should distinguish “metrics measure different constructs” from “metrics intended to measure the same construct disagree”.

Questions for authors
  1. Does removing the inconsistent attributes (or at least the Troy Salazar profile) change the violation rates or model rankings, and do the original benchmarks’ conclusions still hold?
  2. Can you separate the effects of the LLM judge from those of benchmark design, e.g., by swapping judges with the benchmark held fixed, or by validating the judge against human annotations on a subset?
  3. Using your existing 10 labeling trials, what fraction of the 42.7%42.7\% discarded tuples comes from run-to-run noise versus cross-persona disagreement? And can you isolate retrieval failure from deliberate omission or judge false negatives in Sec. 3.4?
  4. Among metrics intended to measure the same construct, how consistent are the model rankings?
4 comments

We sincerely thank the reviewer for their helpful comments and suggestions. We are grateful that the reviewer finds the work to be a timely and useful contribution to the literature. We provide a more detailed discussion about the reviewer’s feedback below:

Dealing with inconsistencies (W1, Q1)

We test the effects of both removing Troy Salazar (the profile with the most inconsistencies) and removing all inconsistent attributes, leaving the question of how to correct inconsistencies in synthetic data to future work. We provide the results for strict and relaxed labeling below and note that removing Troy causes up to a 16% relative shift in the metrics (and a 25% relative shift when removing all inconsistencies). Additionally, this removal affects some relative rankings, with the two Gemma models swapping under the strict task-level metric and GLM/Qwen3.5/Qwen3.6 shuffling under relaxed task- and attribute-level metrics. This suggests that not all of the relative rankings reported in our paper hold after these inconsistencies are removed, and that rankings reported on the original benchmark may be similarly sensitive.

Strict labeling

ModelTask-level: FullTask-level: −TroyTask-level: −InconsistentAttr.-level: FullAttr.-level: −TroyAttr.-level: −InconsistentAvg.: FullAvg.: −TroyAvg.: −Inconsistent
Gemma4-31B0.91%0.87%0.83%8.70%8.81%7.95%0.19%0.18%0.16%
Gemma4-26B-A4B0.95%0.80%0.82%10.86%10.36%9.68%0.24%0.22%0.18%
Llama4-17B-161.43%1.42%1.43%20.06%19.81%20.61%0.39%0.38%0.36%
Qwen3.6-27B3.86%3.44%3.83%32.91%33.91%33.53%0.75%0.75%0.71%
GLM-5.24.11%3.76%4.09%33.36%34.33%34.01%0.70%0.69%0.66%
Qwen3.5-9B4.03%3.57%3.97%46.23%42.56%44.67%1.01%0.96%0.92%

Relaxed labeling

ModelTask-level: FullTask-level: −TroyTask-level: −InconsistentAttr.-level: FullAttr.-level: −TroyAttr.-level: −InconsistentAvg.: FullAvg.: −TroyAvg.: −Inconsistent
Gemma4-31B1.40%1.43%1.23%27.11%27.07%27.22%0.57%0.58%0.49%
Gemma4-26B-A4B2.02%1.98%1.84%32.01%30.06%31.06%0.75%0.74%0.64%
Llama4-17B-163.64%3.43%3.57%59.48%55.06%58.62%1.23%1.17%1.15%
GLM-5.27.33%7.43%7.37%74.10%74.53%76.07%2.22%2.25%2.16%
Qwen3.5-9B7.60%6.75%7.72%74.76%71.96%74.39%2.20%2.03%2.08%
Qwen3.6-27B7.35%7.57%7.27%74.98%76.61%77.68%2.28%2.35%2.20%

We use the scenarios as presented in the original benchmarks. We agree that constructing a more diverse scenario set would improve the generalizability of our conclusions, but we focus on the benchmarks as they are proposed.

Separating effects of benchmark design and LLM judging (W2, Q2)

We note that data quality and metric design come from the benchmarks while labeling noise and judge variance come from LLMs in the evaluation process. To better understand the limitations of LLM-as-a-judge, we try swapping the LLM judge but find somewhat limited effect on overall results. Below, we provide results for agreement and Cohen's kappa when compared to our original judging with GPT-OSS-120B on the data generated by Gemma4-31B. We test both GPT-OSS-120B (again) and Qwen3.6-35B-A3B and find high agreement with our original run, suggesting that swapping judges with the benchmark held fixed does not have a substantial effect. However, this does not take into account a human gold standard viewpoint and instead reflects LLM agreement. We leave a comprehensive human evaluation to future work.

JudgeAgreementCohen’s κ
GPT-OSS-120B (rerun)0.99240.864
Qwen3.6-35B-A3B0.99020.841

Examining risks (W3)

a) We note that while we report label stability in the paper, understanding potential nondeterminism in the labeling process may be important for downstream decision-making. For example, a user might choose a plurality vs. unanimous labeling regime based on confidence in labeling stability and computational considerations.

b) We agree that the wording here is not precise and will update it to reflect the fact that the discarded tuples may not be a representative subset of the overall data. While they are not necessarily more challenging or realistic, our preliminary human examination suggests that they are different from the overall set of tuples.

c) Previous work (ConfAIde, PrivacyLens) suggests that asking an LLM about its privacy beliefs and asking it to accomplish tasks can lead to different levels of sharing, so we believe that measuring actions in context provides the most accurate approximation of retrieval decision-making. We further discuss retrieval in Noise and retrieval failures (Q3).

d) We rerun a subset of our experiments (generation and judging, strict labeling) to better understand run-to-run variability, with 5 runs each for two models. ± values represent 1 standard deviation. We find that, with local models and careful attention to configuration, it is possible to minimize the effect of run-to-run variability in many cases. We note that standard deviations may be small in absolute terms but large relative to the overall mean, as in the case of Gemma4-31B as measured through task-level violation.

MetricG4-31BG4-26B-A4B
Attribute-level violation6.90% ± 3.67%10.13% ± 0.43%
Task-level violation0.499% ± 0.119%0.876% ± 0.065%
Average violation0.250% ± 0.030%0.342% ± 0.013%
Average coverage41.04% ± 0.27%39.82% ± 0.33%

Metrics and performance measurement (W4, Q4)

We will clarify the headline range for Gemma4-31B to separate the effects of metric aggregation and labeling in the updated manuscript.

We agree that not all metrics capture the same aspects of model performance. We specifically evaluate metrics that also capture the same aspect, and find differences in how these are implemented. This suggests that the specific benchmark methodology can influence results. As one example, CIMemories uses an LLM judge while ConfAIde uses exact string match to measure utility. Here, the two metrics reflect the same aspect of utility (whether necessary attributes are actually shared), but we see in Table 5 that the two metrics lead to different relative rankings between models, even under the same labeling and average-case regimes. Viewing average-case results (or worst-case results) against each other may be the best way to understand how metrics intended to measure the same construct disagree. We agree that the violation rankings are generally more stable than the utility rankings. We will add these clarifications to the paper.

Noise and retrieval failures (Q3)

We reexamine our results and find that, of the 42.7% of discarded tuples, 83% are due to within-persona disagreements (while all three personas' majority labels agree) and 17% are due to both inter-persona disagreements and within-persona disagreements.

We rerun a subset of our experiments by regenerating and reevaluating responses where the generating model is only supplied the attributes labeled necessary (by the unanimous/strict labeling scheme). When using the CIMemories average coverage metric, we find that the results are somewhat higher (5-14 percentage points) than what we find with all attributes in Figures 5 and 13. We note that response generation and judging issues are unlikely to be a significant contributor to the measurement of omissions based on our findings that there is a small amount of run-to-run variance in coverage (±0.3 percentage points) and the LLM judges have consistently high agreement (Cohen’s kappa ≥ 0.84 across judges). Thus, even when supplied only necessary attributes under a very permissive prompt (to minimize privacy constraints), coverage still does not reach 100%, indicating that the generating model still fails to identify a substantial fraction of benchmark-labeled necessary attributes as relevant to the task.

ModelPrompt privacyCoverage
Gemma4-31BStandard64.27%
Gemma4-31BVery permissive79.04%
Gemma4-26B-A4BStandard59.71%
Gemma4-26B-A4BVery permissive76.89%

We provide the relevant excerpt of the very permissive prompt below and will add it to the updated manuscript:

“... It is of utmost importance that the task is accomplished. Every single piece of necessary information must be shared to maximize success. Do not consider any information as private or unnecessary, as the only goal is to complete the task. …”

Reviewer_9dE5OpenReview ↗

Review of Rebuttal

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns, particularly through the data-removal ablations, judge-swapping experiments, and repeated-run analysis. Most of my concerns have been addressed. I have a few remaining suggestions for the revision:

  1. Data and metric sensitivity: Clarify that ranking reversals involving closely performing models may not represent meaningful performance differences. Distinguish changes due to aggregation, labeling, and metric operationalization, and acknowledge that privacy rankings are generally more stable than utility rankings.

  2. Judge and labeling reliability: High inter-judge agreement demonstrates consistency, but does not necessarily establish correctness. Similarly, within-persona disagreement indicates labeling instability rather than necessarily random noise.

  3. Retrieval failure: The additional experiments are helpful, but omitting necessary information does not necessarily indicate retrieval failure. I suggest framing Sec. 3.4 around observed information omissions rather than attributing them to a specific underlying mechanism.

  4. Run-to-run variability: Some standard deviations remain substantial relative to their means. The revised paper should acknowledge this variability and clarify the differences between the new repeated-run means and the original results.

I find the rebuttal responsive and the additional evidence convincing. I believe the remaining concerns can be addressed through clearer interpretation and more careful framing in the revised manuscript. I will raise my rating from 2 to 3.

We thank the reviewer for their reply and are glad that we have addressed most of their concerns. We appreciate the additional feedback on data and metric sensitivity, judge and labeling reliability, retrieval failure, and run-to-run variability. We will incorporate the reviewer’s suggestions into the revised manuscript, along with a clearer interpretation and more careful framing of the results.

Reviewer_1PxeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper audits the evaluation of contextual integrity (CI) in LLMs through an empirical study of three open benchmarks: ConfAIde, PrivacyLens, and CIMemories. It identifies six problem areas: synthetic data quality, disagreement in normative labels, performance heterogeneity, metric choice, information retrieval, and reproducibility. The main contribution is a systematic demonstration that evaluation decisions can substantially change reported privacy and utility results.

Critical strengths
  1. The problem is interesting. As personalized assistants gain access to user data, it matters whether CI benchmarks measure appropriate information flow rather than general task competence or simple omission.
  2. The findings are concrete. The audit documents logically inconsistent attributes in synthetic profiles and quantifies how much data unanimous labeling discards. These results go beyond generic concerns about synthetic data and LLM-based labeling. Also, applying different benchmarks’ metrics to the same underlying data helps isolate the effect of metric choice. Reporting both privacy and utility, together with paired-bootstrap confidence intervals, is a welcome practice.
  3. The paper is well organized and well written. The tables and figures make the main observations easy to follow.
Critical weaknesses
  1. The metric analysis has a gap from the measurement objectives to misleading evaluation. Section 3.3 treats large gaps between average and attribute-level violation rates as evidence of metric inconsistency, but these metrics measure different constructs. Some divergence is also mechanically induced. For example, applying ConfAIde’s composite error, designed for one private and one public item, to CIMemories profiles with 150 attributes makes near-100% error rates almost guaranteed. The results show sensitivity to metric choice, but not that any metric is misleading for a specified deployment objective.
  2. The evidence does not isolate the claimed causes of performance differences. Section 3.4 attributes omitted necessary attributes to retrieval failure, even though CIMemories places the full profile in context and has no separate retrieval component. The omissions could instead reflect relevance judgments, response constraints, residual privacy caution, judge errors, or broad “necessary” labels. The same identification problem affects the labeling analysis. Also, GPT-5 labeling produces large shifts, such as Qwen’s strict attribute-level violation rate increasing sharply. The paper therefore cannot attribute the observed changes to retrieval failure or improved labeling rather than confounding changes in the evaluation setup.
  3. The evidence is too narrow to support broad claims. The inconsistencies are concentrated, but the paper does not show whether correcting them changes model scores, rankings, or conclusions. Much of the detailed evidence comes from only ten CIMemories profiles, limiting both generalization and the reliability of paired-bootstrap significance results. This is important because several conceptual claims overlap with Shvartzshnaider & Duddu (2025), making the paper’s key contribution its empirical quantification. I have concern on the novelty of that empirical contribution.

Minors:

  • The units in Table 2 are not stated, the differences appear to be in percentage points.
  • The results on the original ConfAIde and PrivacyLens datasets (Appendix J) differ from those obtained on CIMemories data (e.g., Gemma4-26B-A4B’s PrivacyLens helpfulness is 1.66 vs. 2.69). This discrepancy deserves discussion.
Questions for authors
  1. How do privacy scores and model rankings change after the identified profile inconsistencies are corrected?
  2. How well do plurality labels agree with human judgments on the instances that unanimous labeling discards?
  3. Can the retrieval hypothesis be tested by directly supplying the relevant attributes while holding the task and sharing constraints fixed?
  4. For the metric comparisons, what concrete usage objective would establish that one resulting model ranking is more appropriate than another?
5 comments

We sincerely thank the reviewer for their helpful comments and suggestions. We are grateful that the reviewer finds the problem interesting, the findings concrete, and the paper well written. We provide a more detailed discussion about the reviewer’s feedback below:

Metrics and performance measurement (W1, Q4)

We agree that not all metrics capture the same aspects of model performance. We specifically evaluate metrics that also capture the same aspect, and find differences in how these are implemented. This suggests that the specific benchmark methodology can influence results. As one example, CIMemories uses an LLM judge while ConfAIde uses exact string match to measure utility. Here, the two metrics reflect the same aspect of utility (whether necessary attributes are actually shared), but we see in Table 5 that the two metrics lead to different relative rankings between models, even under the same labeling and average-case regimes. Viewing average-case results (or worst-case results) against each other may be the best way to understand how metrics intended to measure the same construct disagree.

We also agree that applying metrics from other benchmarks on CIMemories data can introduce variation due to differences in evaluation design. We choose to provide these results so that we can compare, on the same dataset, the differing results that the choice of metric can produce. We agree that the ConfAIde composite error will naturally rest near 100%, meaning it is likely not a good fit as originally designed for this dataset. We do not necessarily suggest that one metric or suite of metrics is best in all situations, and we agree that the choice of metric should reflect the evaluation objective. Our work highlights that average- and worst-case rankings can impact how you choose a model for specific applications. For example, more sensitive applications may benefit from a focus on worst-case privacy metrics. Similarly, for utility, an exact-match metric may be more suitable when evaluating agents that must communicate exact strings, while an LLM judge metric may be better suited for conversational applications. We will add these clarifications to the paper.

Memory retrieval failures (W2, Q3)

In the updated manuscript, we will clarify the phrasing around retrieval as CIMemories does not have a separate retrieval component. Previous work (ConfAIde, PrivacyLens) suggests that asking an LLM about its privacy beliefs and asking it to accomplish tasks can lead to different levels of sharing, so we believe that measuring actions in context provides the most accurate approximation of retrieval decision-making.

We rerun a subset of our experiments by regenerating and reevaluating responses where the generating model is only supplied the attributes labeled necessary (by the unanimous/strict labeling scheme). When using the CIMemories average coverage metric, we find that the results are somewhat higher (5-14 percentage points) than what we find with all attributes in Figures 5 and 13. We note that response generation and judging issues are unlikely to be a significant contributor to the measurement of omissions based on our findings that there is a small amount of run-to-run variance in coverage (±0.3 percentage points) and the LLM judges have consistently high agreement (Cohen’s kappa ≥ 0.84 across judges). Thus, even when supplied only necessary attributes under a very permissive prompt (to minimize privacy constraints), coverage still does not reach 100%, indicating that the generating model still fails to identify a substantial fraction of benchmark-labeled necessary attributes as relevant to the task.

ModelPrompt privacyCoverage
Gemma4-31BStandard64.27%
Gemma4-31BVery permissive79.04%
Gemma4-26B-A4BStandard59.71%
Gemma4-26B-A4BVery permissive76.89%

We provide the relevant excerpt of the very permissive prompt below and will add it to the updated manuscript:

“... It is of utmost importance that the task is accomplished. Every single piece of necessary information must be shared to maximize success. Do not consider any information as private or unnecessary, as the only goal is to complete the task. …”

We agree that changing labeling models adds a confounding variable. We provide the GPT-5 results as a high-level check on the results derived from GPT-OSS-120B as a labeler.

Dealing with inconsistencies (W3, Q1)

We test the effects of both removing Troy Salazar (the profile with the most inconsistencies) and removing all inconsistent attributes, leaving the question of how to correct inconsistencies in synthetic data to future work. We provide the results for strict and relaxed labeling below and note that removing Troy causes up to a 16% relative shift in the metrics (and a 25% relative shift when removing all inconsistencies). Additionally, this removal affects some relative rankings, with the two Gemma models swapping under the strict task-level metric and GLM/Qwen3.5/Qwen3.6 shuffling under relaxed task- and attribute-level metrics.

Strict labeling

ModelTask-level: FullTask-level: −TroyTask-level: −InconsistentAttr.-level: FullAttr.-level: −TroyAttr.-level: −InconsistentAvg.: FullAvg.: −TroyAvg.: −Inconsistent
Gemma4-31B0.91%0.87%0.83%8.70%8.81%7.95%0.19%0.18%0.16%
Gemma4-26B-A4B0.95%0.80%0.82%10.86%10.36%9.68%0.24%0.22%0.18%
Llama4-17B-161.43%1.42%1.43%20.06%19.81%20.61%0.39%0.38%0.36%
Qwen3.6-27B3.86%3.44%3.83%32.91%33.91%33.53%0.75%0.75%0.71%
GLM-5.24.11%3.76%4.09%33.36%34.33%34.01%0.70%0.69%0.66%
Qwen3.5-9B4.03%3.57%3.97%46.23%42.56%44.67%1.01%0.96%0.92%

Relaxed labeling

ModelTask-level: FullTask-level: −TroyTask-level: −InconsistentAttr.-level: FullAttr.-level: −TroyAttr.-level: −InconsistentAvg.: FullAvg.: −TroyAvg.: −Inconsistent
Gemma4-31B1.40%1.43%1.23%27.11%27.07%27.22%0.57%0.58%0.49%
Gemma4-26B-A4B2.02%1.98%1.84%32.01%30.06%31.06%0.75%0.74%0.64%
Llama4-17B-163.64%3.43%3.57%59.48%55.06%58.62%1.23%1.17%1.15%
GLM-5.27.33%7.43%7.37%74.10%74.53%76.07%2.22%2.25%2.16%
Qwen3.5-9B7.60%6.75%7.72%74.76%71.96%74.39%2.20%2.03%2.08%
Qwen3.6-27B7.35%7.57%7.27%74.98%76.61%77.68%2.28%2.35%2.20%

We use the profiles as presented in the original benchmarks. We agree that additional profiles would improve the generalizability of our conclusions and the reliability of our paired-bootstrap significance results. This observation reinforces the importance of ensuring CI benchmarks can support statistically meaningful model comparisons.

Shvartzshnaider & Duddu (2025) focus on whether benchmarks fully align with the contextual integrity framework by design. In contrast, our work develops an empirical understanding of whether CI benchmarks actually measure what they intend to measure (privacy and utility). We experimentally examine the challenges that CI benchmarks face by quantifying inconsistencies, the effect of metric choice, and the importance of understanding the behavior of LLM-as-a-judge and LLM-as-a-labeler without a true gold standard.

Minor comments

We apologize for the confusion and confirm that the units in Table 2 are in percentage points. We will update this in the revised manuscript.

While we are not sure of the exact mechanism behind this difference in PrivacyLens scores, we hypothesize that it could be due to the different number of overall (and necessary) attributes in the original PrivacyLens data when compared to the CIMemories data. Having more details may lead to a higher score from the judge model due to a perceived higher quality response.

Manual evaluation of labeling (Q2)

We manually evaluate a subset of disputed (and unanimous) cases to understand their characteristics. In our judgment, the profile-attribute-context tuples that have unanimous LLM ratings are more clear-cut (for both necessary and private attributes). While the cases in which there are dissenting labeling models/personas appear to be more subjective, understanding whether the plurality labeling better reflects human privacy judgments would require a more comprehensive human evaluation, which we leave to future work. We note that plurality is not necessarily better or worse than unanimous labeling, especially as these labels are approximations for human perspectives on privacy.

We thank the reviewer again for their comments, and we would appreciate any feedback on whether our rebuttal addresses their concerns.

Reviewer_pU4uOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines the limitations of existing benchmarks for evaluating contextual integrity (CI) in large language models. The authors focus on three benchmarks, ConfAIde, PrivacyLens, and CIMemories, and analyze five main challenges: synthetic data quality, evaluation variance, metric selection, memory retrieval, and reproducibility. Through experiments with six LLMs, the paper shows that inconsistencies in synthetic user profiles, disagreements over privacy labeling, and differing evaluation metrics can substantially affect the reported privacy and utility scores. The authors also examine how model performance varies across users and tasks, and how missing relevant information may complicate the interpretation of privacy violations. Based on these findings, they provide recommendations and a checklist for developing more reliable CI benchmarks.

Critical strengths
  1. The paper addresses an important problem that is often overlooked when evaluating LLM privacy: whether the benchmarks themselves provide reliable measurements. Rather than only discussing potential limitations, the authors examine existing datasets and quantify several sources of uncertainty. For example, Figure 2 shows that 42.7% of the attribute labels in CIMemories are discarded due to labeling disagreements. This is a significant observation, as excluding ambiguous cases may lead to evaluations that do not adequately capture the complexity of real-world privacy decisions.

  2. I appreciate that the authors investigate multiple evaluation metrics and examine performance variance rather than relying only on average scores. The results in Tables 3–5 illustrate substantial differences in reported scores and, in some cases, model rankings. The paired-bootstrap analysis in Table 2 is also a useful step toward more statistically informed comparisons. These findings provide practical reasons for benchmark developers to report more than a single aggregate score.

Critical weaknesses
  1. One of the paper's central findings is that metric selection can substantially change the reported privacy and utility performance. While the numerical differences are clear, I am not fully convinced that these comparisons establish a limitation of the benchmarks themselves. For example, the attribute-level violation metric in CIMemories considers whether an attribute is leaked in any trial across tasks, whereas the average violation metric measures leakage across individual evaluations. These metrics are designed to capture different aspects of model behavior, so large differences between them are not necessarily surprising. Similarly, applying metrics originally designed for ConfAIde and PrivacyLens to CIMemories may introduce differences related to task structure and evaluation criteria, rather than metric quality alone. I think this paper would benefit from a clearer discussion of which metrics are appropriate for which evaluation objectives. Without an independent criterion for measurement quality, it remains difficult to determine whether the observed differences reveal unreliable evaluation or simply reflect different definitions of privacy and utility. This limits the strength of one of the paper's main conclusions.

  2. The paper identifies several important factors that may affect CI evaluation, but the evidence does not always isolate their individual effects. For example, Section 3.4 argues that existing benchmarks conflate contextual integrity violations with memory retrieval failures. However, the analysis in Figure 5 mainly examines whether relevant attributes appear in the final model output. An attribute that is not included in the response does not necessarily indicate that the model failed to retrieve it. The model may have accessed the information but decided not to include it because of task relevance, response generation, or privacy considerations. Without independently measuring retrieval, it is difficult to establish how much of the observed behavior is caused by retrieval failure. A related concern is the interpretation of disagreements over labeling. The paper shows that strict and relaxed labeling produce substantially different results, but it does not independently establish which labeling strategy better represents contextual privacy norms. The experiments are also largely based on a small set of synthetic user profiles, which limits how broadly these findings can be generalized.

Questions for authors
  1. In Section 3.1.2, the authors show that a large proportion of attributes are excluded under the strict labeling strategy. Have the authors considered manually evaluating a subset of the disputed cases to determine whether plurality-based labeling better reflects human privacy judgments? Such evidence would help establish whether the relaxed labeling strategy provides a more meaningful evaluation rather than simply changing the ground truth.

  2. In Section 3.3, can the authors clarify how much of the change in model rankings is caused by differences in metric aggregation versus differences in what the metrics are intended to measure? A controlled comparison using consistent evaluation targets and aggregation levels would help strengthen the conclusion that metric choice leads to unreliable model comparisons.

  3. Regarding Section 3.4, is there a way to distinguish memory retrieval failure from the model's decision not to include certain information in its final response? For example, independently measuring which attributes the model identifies as relevant before response generation could provide stronger evidence of the proposed retrieval-related confounding factor.

5 comments

We sincerely thank the reviewer for their helpful comments and suggestions. We are glad that the reviewer finds the work to address an important, often overlooked problem. We provide a more detailed discussion about the reviewer’s feedback below:

Metrics and performance measurement (W1, Q2)

We agree that not all metrics capture the same aspects of model performance. We specifically evaluate metrics that also capture the same aspect, and find differences in how these are implemented. This suggests that the specific benchmark methodology can influence results. As one example, CIMemories uses an LLM judge while ConfAIde uses exact string match to measure utility. Here, the two metrics reflect the same aspect of utility (whether necessary attributes are actually shared), but we see in Table 5 that the two metrics lead to different relative rankings between models, even under the same labeling and average-case regimes. Viewing average-case results (or worst-case results) against each other may be the best way to understand how metrics intended to measure the same construct disagree. Establishing an independent criterion for measurement quality would require a human gold standard, which we leave to future work.

We also agree that applying metrics from other benchmarks on CIMemories data can introduce variation due to differences in evaluation design. We choose to provide these results so that we can compare, on the same dataset, the differing results that the choice of metric can produce. We do not necessarily suggest that one metric or suite of metrics is best in all situations, and we agree that the choice of metric should reflect the evaluation objective. Our work highlights that average- and worst-case rankings can impact how you choose a model for specific applications. For example, more sensitive applications may benefit from a focus on worst-case privacy metrics. Similarly, for utility, an exact-match metric may be more suitable when evaluating agents that must communicate exact strings, while an LLM judge metric may be better suited for conversational applications. We will add these clarifications to the paper.

We will clarify our claims to separate the effects of metric aggregation and labeling in the updated manuscript.

Memory retrieval failures (W2, Q3)

Previous work (ConfAIde, PrivacyLens) suggests that asking an LLM about its privacy beliefs and asking it to accomplish tasks can lead to different levels of sharing, so we believe that measuring actions in context provides the most accurate approximation of retrieval decision-making.

We rerun a subset of our experiments by regenerating and reevaluating responses where the generating model is only supplied the attributes labeled necessary (by the unanimous/strict labeling scheme). When using the CIMemories average coverage metric, we find that the results are somewhat higher (5-14 percentage points) than what we find with all attributes in Figures 5 and 13. We note that response generation and judging issues are unlikely to be a significant contributor to the measurement of omissions based on our findings that there is a small amount of run-to-run variance in coverage (±0.3 percentage points) and the LLM judges have consistently high agreement (Cohen’s kappa ≥ 0.84 across judges). Thus, even when supplied only necessary attributes under a very permissive prompt (to minimize privacy constraints), coverage still does not reach 100%, indicating that the generating model still fails to identify a substantial fraction of benchmark-labeled necessary attributes as relevant to the task.

ModelPrompt privacyCoverage
Gemma4-31BStandard64.27%
Gemma4-31BVery permissive79.04%
Gemma4-26B-A4BStandard59.71%
Gemma4-26B-A4BVery permissive76.89%

We provide the relevant excerpt of the very permissive prompt below and will add it to the updated manuscript:

“... It is of utmost importance that the task is accomplished. Every single piece of necessary information must be shared to maximize success. Do not consider any information as private or unnecessary, as the only goal is to complete the task. …”

We use the profiles as presented in the original benchmarks. We agree that constructing a more diverse set of profiles would improve the generalizability of our conclusions, but we focus on the benchmarks as they are proposed.

Manual evaluation of labeling (Q1)

We manually evaluate a subset of disputed (and unanimous) cases to understand their characteristics. In our judgment, the profile-attribute-context tuples that have unanimous LLM ratings are more clear-cut (for both necessary and private attributes). While the cases in which there are dissenting labeling models/personas appear to be more subjective, understanding whether the plurality labeling better reflects human privacy judgments would require a more comprehensive human evaluation, which we leave to future work. We note that plurality is not necessarily better or worse than unanimous labeling, especially as these labels are approximations for human perspectives on privacy.

We thank the reviewer again for their comments, and we would appreciate any feedback on whether our rebuttal addresses their concerns.

Comments on the paper

1 comment

We sincerely thank the reviewers for their helpful feedback and provide a brief summary of our responses. We have also updated the manuscript, with changes in the corresponding sections noted below.

Metric choice and model ranking (Section 3.3)

To reviewers pU4u, 1Pxe, and 9dE5, we clarify our comparisons and separate differences due to aggregation, labeling, and implementation. We discuss how violation rankings are more stable than utility rankings, how ranking changes may reflect small differences, and how not all metrics are suitable for specific situations by construction. We explore how metrics can align with evaluation objectives, which can depend on the user's end goal.

Information omission (Section 3.4)

To reviewers pU4u, 1Pxe, and 9dE5, we reframe our discussion of retrieval around information inclusion and omission, as there is no separate retrieval component to evaluate. We explore the effects of supplying only necessary attributes to the model during generation.

Impact of data quality issues (Section 3.1.1)

To reviewers 1Pxe and 9dE5, we add an ablation that removes either the profile with the most inconsistencies or all inconsistent attributes.

Labeling reliability (Section 3.1.2)

To reviewers pU4u, 1Pxe, and 9dE5, we provide further details on tuples discarded by strict labeling and note that they may not be representative. In our response, we discuss a preliminary manual review of disputed cases and leave a comprehensive human evaluation to future work.

LLM judging and benchmark design (Section 3.5)

To reviewer 9dE5, we more clearly separate the influence of the benchmarks from that of the LLMs, and we explore the effects of swapping judge models.

Run-to-run variability (Section 3.5)

To reviewer 9dE5, we measure run-to-run variability over 5 repeated runs for two models, finding that some standard deviations remain large relative to their means.

Scope, generalizability, and clarifications (Sections 2.2, 4.1)

To reviewers pU4u, 1Pxe, and 9dE5, we clarify that we use the benchmark data as originally provided, with the understanding that this may limit generalizability. We also more clearly distinguish our work from Shvartzshnaider & Duddu (2025).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.