- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents a meta-evaluation of three benchmarks grounded in contextual integrity (CI) with six open-weight models. The analysis reveals five challenges, including data quality, variance, metric choice, CI judgement, and reproducibility. It concludes with a checklist for benchmark designers and users.
Critical strengths
- This is a timely work. CI benchmarks are increasingly cited in the studies of agent privacy behavior, so it is necessary to scrutinize the measurement validity.
- Identifying and quantifying the inconsistency and limited diversity in synthetic data is a useful contribution.
- Calling for the attention to statistical rigor is a necessary and helpful practice.
Critical weaknesses
1. Data quality findings lack an impact analysis
The paper documents inconsistency and limited diversity in the synthetic data, yet an important missing piece is further tests on the impact of such data quality issue, e.g., whether fixing these issues changes the conclusions in the original benchmark paper.
Specifically, Fig. 1 shows one profile (Troy Salazar) accounts for misaligned attributes. So a straightforward ablation would be removing the inconsistent profiles (or at least the most affected profile) and recomputing privacy violation rates in the benchmarks. Moreover, constructing a more diverse scenario set would test whether conclusions are stable under greater diversity.
2. Benchmark-specific issues are conflated with general LLM-as-judge limitations
Some identified problems are inherent to any use of LLMs for labeling and judging, while others stem from specific benchmark design choices. The current analysis does not separate them, so readers cannot tell where the problems actually originate.
3. Several risks are asserted rather than demonstrated.
(a) labeler nondeterminism is raised as a concern, yet the authors ran 10 labeling trials and could directly report label stability;
(b) the claim that discarded tuples represent “more challenging and realistic” cases is not supported beyond the observation that violation rates rise under relaxed labeling;
(c) the interpretation of unshared necessary attributes as retrieval failure (Sec. 3.4) is not isolated from alternative explanations such as deliberate omission, over-inclusive labels, or judge false negatives;
(d) run-to-run variability is discussed (Sec. 3.5) but not measured in the authors’ own experiments.
4. The metric comparison partly reflects definitional differences
The headline range for Gemma4-31B spans both average vs. worst-case aggregation and strict vs. relaxed labeling. Worst-case-over-trials metrics will exceed averages by construction. Relaxed labeling adds ambiguous tuples and changes the denominator. That these numbers differ is expected.
The more meaningful question is whether metrics intended to measure the same construct disagree on rankings. Appendix F suggests the violation rankings are in fact quite stable (Gemma4-31B > Gemma4-26B > Llama4 under nearly every metric). The paper should distinguish “metrics measure different constructs” from “metrics intended to measure the same construct disagree”.
Questions for authors
- Does removing the inconsistent attributes (or at least the Troy Salazar profile) change the violation rates or model rankings, and do the original benchmarks’ conclusions still hold?
- Can you separate the effects of the LLM judge from those of benchmark design, e.g., by swapping judges with the benchmark held fixed, or by validating the judge against human annotations on a subset?
- Using your existing 10 labeling trials, what fraction of the discarded tuples comes from run-to-run noise versus cross-persona disagreement? And can you isolate retrieval failure from deliberate omission or judge false negatives in Sec. 3.4?
- Among metrics intended to measure the same construct, how consistent are the model rankings?
4 comments
Examining risks (W3)
a) We note that while we report label stability in the paper, understanding potential nondeterminism in the labeling process may be important for downstream decision-making. For example, a user might choose a plurality vs. unanimous labeling regime based on confidence in labeling stability and computational considerations.
b) We agree that the wording here is not precise and will update it to reflect the fact that the discarded tuples may not be a representative subset of the overall data. While they are not necessarily more challenging or realistic, our preliminary human examination suggests that they are different from the overall set of tuples.
c) Previous work (ConfAIde, PrivacyLens) suggests that asking an LLM about its privacy beliefs and asking it to accomplish tasks can lead to different levels of sharing, so we believe that measuring actions in context provides the most accurate approximation of retrieval decision-making. We further discuss retrieval in Noise and retrieval failures (Q3).
d) We rerun a subset of our experiments (generation and judging, strict labeling) to better understand run-to-run variability, with 5 runs each for two models. ± values represent 1 standard deviation. We find that, with local models and careful attention to configuration, it is possible to minimize the effect of run-to-run variability in many cases. We note that standard deviations may be small in absolute terms but large relative to the overall mean, as in the case of Gemma4-31B as measured through task-level violation.
| Metric | G4-31B | G4-26B-A4B |
|---|---|---|
| Attribute-level violation | 6.90% ± 3.67% | 10.13% ± 0.43% |
| Task-level violation | 0.499% ± 0.119% | 0.876% ± 0.065% |
| Average violation | 0.250% ± 0.030% | 0.342% ± 0.013% |
| Average coverage | 41.04% ± 0.27% | 39.82% ± 0.33% |
Metrics and performance measurement (W4, Q4)
We will clarify the headline range for Gemma4-31B to separate the effects of metric aggregation and labeling in the updated manuscript.
We agree that not all metrics capture the same aspects of model performance. We specifically evaluate metrics that also capture the same aspect, and find differences in how these are implemented. This suggests that the specific benchmark methodology can influence results. As one example, CIMemories uses an LLM judge while ConfAIde uses exact string match to measure utility. Here, the two metrics reflect the same aspect of utility (whether necessary attributes are actually shared), but we see in Table 5 that the two metrics lead to different relative rankings between models, even under the same labeling and average-case regimes. Viewing average-case results (or worst-case results) against each other may be the best way to understand how metrics intended to measure the same construct disagree. We agree that the violation rankings are generally more stable than the utility rankings. We will add these clarifications to the paper.
Noise and retrieval failures (Q3)
We reexamine our results and find that, of the 42.7% of discarded tuples, 83% are due to within-persona disagreements (while all three personas' majority labels agree) and 17% are due to both inter-persona disagreements and within-persona disagreements.
We rerun a subset of our experiments by regenerating and reevaluating responses where the generating model is only supplied the attributes labeled necessary (by the unanimous/strict labeling scheme). When using the CIMemories average coverage metric, we find that the results are somewhat higher (5-14 percentage points) than what we find with all attributes in Figures 5 and 13. We note that response generation and judging issues are unlikely to be a significant contributor to the measurement of omissions based on our findings that there is a small amount of run-to-run variance in coverage (±0.3 percentage points) and the LLM judges have consistently high agreement (Cohen’s kappa ≥ 0.84 across judges). Thus, even when supplied only necessary attributes under a very permissive prompt (to minimize privacy constraints), coverage still does not reach 100%, indicating that the generating model still fails to identify a substantial fraction of benchmark-labeled necessary attributes as relevant to the task.
| Model | Prompt privacy | Coverage |
|---|---|---|
| Gemma4-31B | Standard | 64.27% |
| Gemma4-31B | Very permissive | 79.04% |
| Gemma4-26B-A4B | Standard | 59.71% |
| Gemma4-26B-A4B | Very permissive | 76.89% |
We provide the relevant excerpt of the very permissive prompt below and will add it to the updated manuscript:
“... It is of utmost importance that the task is accomplished. Every single piece of necessary information must be shared to maximize success. Do not consider any information as private or unnecessary, as the only goal is to complete the task. …”
Review of Rebuttal
Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns, particularly through the data-removal ablations, judge-swapping experiments, and repeated-run analysis. Most of my concerns have been addressed. I have a few remaining suggestions for the revision:
-
Data and metric sensitivity: Clarify that ranking reversals involving closely performing models may not represent meaningful performance differences. Distinguish changes due to aggregation, labeling, and metric operationalization, and acknowledge that privacy rankings are generally more stable than utility rankings.
-
Judge and labeling reliability: High inter-judge agreement demonstrates consistency, but does not necessarily establish correctness. Similarly, within-persona disagreement indicates labeling instability rather than necessarily random noise.
-
Retrieval failure: The additional experiments are helpful, but omitting necessary information does not necessarily indicate retrieval failure. I suggest framing Sec. 3.4 around observed information omissions rather than attributing them to a specific underlying mechanism.
-
Run-to-run variability: Some standard deviations remain substantial relative to their means. The revised paper should acknowledge this variability and clarify the differences between the new repeated-run means and the original results.
I find the rebuttal responsive and the additional evidence convincing. I believe the remaining concerns can be addressed through clearer interpretation and more careful framing in the revised manuscript. I will raise my rating from 2 to 3.
We thank the reviewer for their reply and are glad that we have addressed most of their concerns. We appreciate the additional feedback on data and metric sensitivity, judge and labeling reliability, retrieval failure, and run-to-run variability. We will incorporate the reviewer’s suggestions into the revised manuscript, along with a clearer interpretation and more careful framing of the results.
We sincerely thank the reviewer for their helpful comments and suggestions. We are grateful that the reviewer finds the work to be a timely and useful contribution to the literature. We provide a more detailed discussion about the reviewer’s feedback below:
Dealing with inconsistencies (W1, Q1)
We test the effects of both removing Troy Salazar (the profile with the most inconsistencies) and removing all inconsistent attributes, leaving the question of how to correct inconsistencies in synthetic data to future work. We provide the results for strict and relaxed labeling below and note that removing Troy causes up to a 16% relative shift in the metrics (and a 25% relative shift when removing all inconsistencies). Additionally, this removal affects some relative rankings, with the two Gemma models swapping under the strict task-level metric and GLM/Qwen3.5/Qwen3.6 shuffling under relaxed task- and attribute-level metrics. This suggests that not all of the relative rankings reported in our paper hold after these inconsistencies are removed, and that rankings reported on the original benchmark may be similarly sensitive.
Strict labeling
Relaxed labeling
We use the scenarios as presented in the original benchmarks. We agree that constructing a more diverse scenario set would improve the generalizability of our conclusions, but we focus on the benchmarks as they are proposed.
Separating effects of benchmark design and LLM judging (W2, Q2)
We note that data quality and metric design come from the benchmarks while labeling noise and judge variance come from LLMs in the evaluation process. To better understand the limitations of LLM-as-a-judge, we try swapping the LLM judge but find somewhat limited effect on overall results. Below, we provide results for agreement and Cohen's kappa when compared to our original judging with GPT-OSS-120B on the data generated by Gemma4-31B. We test both GPT-OSS-120B (again) and Qwen3.6-35B-A3B and find high agreement with our original run, suggesting that swapping judges with the benchmark held fixed does not have a substantial effect. However, this does not take into account a human gold standard viewpoint and instead reflects LLM agreement. We leave a comprehensive human evaluation to future work.