- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes CICF, a framework for deciding which Contextual Integrity parameter an LLM should clarify when judging whether an information flow is appropriate. For each parameter, it generates candidate specifications and measures aleatoric uncertainty as the mutual information between the model’s judgment and the candidate used. Clarification is then gated by either prediction entropy or a supervised probe on hidden activations that predicts errors. Across three open models on PrivacyLens+ and ConfAIde+, clarifying the highest-uncertainty parameter outperforms random and lowest-uncertainty selection and matches or beats asking the model directly. The authors further show that gains peak at intermediate coverage and that most of the benefit comes from a single clarification round.
Critical strengths
AU ties the model’s uncertainty to a specific CI parameter, which makes it interpretable and useful beyond clarification. Under the oracle gate, AU-MAX beats Random and AU-MIN in all six configurations, and AU-MIN performing worst confirms the signal carries real information about which parameter to clarify.
Critical weaknesses
- AU-MAX only seems to be significantly better than Self-ID in 2 out of the 6 configurations in Table 1. However, it requires generation and K forward passes per scenario compared to a singular call for Self-ID. This cost/benefit trade-off should be discussed explicitly.
- The proposed method depends heavily on the diversity and quality of the Gemini-generated candidates and on the uniform prior over them. The generator also decides which parameters count as “already specific.” There are no ablations on varying the generator model or validating the quality of the candidates.
- The probe needs labeled data for each model and dataset, and it is only evaluated in-distribution (the authors note it is not meant to transfer). While gates beat the provided prior error-detection methods, the authors compare a supervised probe to label-free baselines. There needs to be supervised baselines as well.
Questions for authors
How does the supervised probe for error detection compare with supervised baselines that would have access to the labeled data?
What happens if you change the model that generates the candidates?
How much more expensive is it to run AU-MAX compared to the Self-ID baseline?
4 comments
Weakness 2 / Question 2: "The proposed method depends heavily on the diversity and quality of the Gemini-generated candidates and on the uniform prior over them. The generator also decides which parameters count as 'already specific.' There are no ablations on varying the generator model or validating the quality of the candidates."
The generator, the label, and the prior change which parameter is picked in individual scenarios, but not the conclusion: AU-max is the best selection rule in every collection and its gain is stable under random priors.
- Generator. The paper's candidates come from Gemini 3.8 Flash, which sees the label. We built two label-free collections: same generator (Flash) and a different one (Gemini 3.1 Pro). AU-max is the best of four rules (AU-min, Random, Self-ID, AU-max) in all 6 settings with Flash and all 10 with Pro (Tables R6, R2). Without the label, switching generator moves the AU-max gain by <0.8 pp on PrivacyLens+; on ConfAIde+, Pro gives 1.2–2.4 pp more.
- Uniform prior, explained. To compute AU, each of the K=5 candidates for a parameter needs a probability of being the true specification. The paper assumes all five are equally likely (weight 1/5 each). To test this, we replaced the equal weights with random weights (drawn uniformly over all ways of splitting 1 among 5 candidates, i.e. Dirichlet(1,…,1), new draw per scenario and parameter), recomputed AU, re-selected the parameter, and repeated 200 times. The AU-max gain changed by at most 0.36 pp on average and by <0.9 pp in every draw, in all 20 settings of Tables R1 and R2.
- Diversity. Candidates are never duplicates and share few words (mean pairwise word Jaccard 0.18–0.25). They are related, not paraphrases: mean embedding cosine 0.63–0.68 (0.32–0.36 for unrelated scenarios); only 0.5–2% of pairs exceed 0.9. In 13–40% of candidate sets the judge's decision differs across the five, the variation AU measures.
- "Already specific." The share of scenario–parameter pairs declared already specific varies by generator (Flash 33% PrivacyLens+ / 74% ConfAIde+; Pro 15% / 51%), yet AU-max is the best rule under both.
Table R6. Oracle-gate ΔAcc (pp) of AU-max. Cols 1 vs 2: same generator, with/without label. Cols 2 vs 3: no label, two generators.
| Dataset | Model | Flash, label-cond. | Flash, label-free | Pro, label-free |
|---|---|---|---|---|
| PrivacyLens+ | Qwen3-32B | 12.03 | 8.82 | 9.61 |
| PrivacyLens+ | Gemma-4 | 8.32 | 6.15 | 6.23 |
| PrivacyLens+ | Qwen3.6-27B | 9.13 | 6.75 | 6.90 |
| ConfAIde+ | Qwen3-32B | 4.52 | 4.15 | 5.33 |
| ConfAIde+ | Gemma-4 | 6.33 | 5.11 | 7.52 |
| ConfAIde+ | Qwen3.6-27B | 3.70 | 2.85 | 5.04 |
One may object that the label leak inflates the paper's numbers. It does raise the gain (e.g. 12.03 → 8.82 pp, Qwen3-32B on PrivacyLens+), but the label-free gains remain positive in all settings and AU-max still ranks first, so the ranking of rules does not rely on the label.
The parameter chosen for a single scenario does depend on the candidates (two collections agree on 48–81% of initial errors); AU measures sensitivity to the specifications actually proposed. We did not run a human validation of candidate quality and will list this as a limitation.
Paper change: the label-free Flash collection, the generator and random-prior ablations, and the diversity statistics go in the appendix; the limitation is stated in the paper.
I appreciate the additional ablations. However, candidate quality is not addressed, as there is no metric measuring the plausibility. Additionally, non-Gemini generators are not utilized for the candidates. I will maintain my score.
We thank the reviewer for the follow-up. We address both remaining concerns with new experiments: an LLM-as-a-judge plausibility evaluation for the candidates, and a candidate generator from a different model family.
1. Plausibility metric (LLM-as-a-judge). 83–91% of candidates are judged plausible, and candidate sets contain 4.7–4.9 distinct interpretations out of 5, for both Gemini and non-Gemini candidates.
To evaluate candidate quality systematically, we use an LLM-as-a-judge protocol with two independent judges from different model families (Claude Sonnet 5.5 and GPT-6.1 Terra). The judges evaluate 150 randomly sampled scenario–parameter pairs per benchmark and generator (3,000 candidates total) and never see the ground-truth label.
Each judge rates a candidate as plausible if it meets three criteria:
- (C0) Compatible: does not contradict any explicit word or constraint in the scenario;
- (C1) Supported by original wording: is a reading the parameter's original wording can carry, without adding norm-relevant facts (such as consent, authorization, or legal obligations) that may bias the judgment by assumption; and
- (C2) Confined to the target parameter: leaves every other CI parameter unchanged, so uncertainty is not misattributed.
To measure diversity, each LLM judge groups the 5 candidates of a set into paraphrase clusters; the number of clusters is the number of distinct interpretations (5 = all distinct, 1 = all paraphrases).
Table R9. Candidate plausibility and diversity evaluated via LLM-as-a-judge.
| Generator | Benchmark | Plausible (Sonnet) | Plausible (Terra) | Plausible (both) | Distinct / 5 (Sonnet · Terra) | Judge agreement |
|---|---|---|---|---|---|---|
| Gemini 3.8 Flash | PrivacyLens+ | 87.3% | 93.7% | 86.1% | 4.69 · 4.87 | 92.0% |
| Gemini 3.8 Flash | ConfAIde+ | 86.2% | 99.1% | 85.1% | 4.66 · 4.79 | 86.3% |
| Claude Sonnet 5.5 | PrivacyLens+ | 85.9% | 91.9% | 83.5% | 4.86 · 4.93 | 89.8% |
| Claude Sonnet 5.5 | ConfAIde+ | 91.1% | 98.7% | 90.5% | 4.89 · 4.93 | 91.6% |
2. Non-Gemini generator. AU-max does not depend on Gemini: with Claude Sonnet 5.5 candidates, its gain is higher in all 10 settings. We regenerated every candidate for both benchmarks with Sonnet (identical prompt, label conditioning, and ) and re-ran the full pipeline with the same five judges. AU-max's gain rises by +0.7 to +2.2 pp on PrivacyLens+ and by +1.9 to +12.2 pp on ConfAIde+ (Table R10); the increase is significant in 9/10 settings (paired Wilcoxon, Holm).
Table R10. Oracle-gate (pp) of AU-max with label-conditioned candidates from Gemini 3.8 Flash (paper) vs. Claude Sonnet 5.5. Same judges and scenarios.
| Dataset | Model | Gemini 3.8 Flash | Claude Sonnet 5.5 |
|---|---|---|---|
| PrivacyLens+ | Qwen3-32B | 12.03 | 13.49 |
| PrivacyLens+ | Gemma-4 | 8.32 | 9.01 |
| PrivacyLens+ | Qwen3.6-27B | 9.13 | 10.63 |
| PrivacyLens+ | GPT-6 Luna | 10.99 | 13.02 |
| PrivacyLens+ | DeepSeek V4 Flash | 11.03 | 13.27 |
| ConfAIde+ | Qwen3-32B | 4.52 | 6.44 |
| ConfAIde+ | Gemma-4 | 6.33 | 12.56 |
| ConfAIde+ | Qwen3.6-27B | 3.70 | 10.81 |
| ConfAIde+ | GPT-6 Luna | 6.78 | 18.96 |
| ConfAIde+ | DeepSeek V4 Flash | 2.19 | 7.04 |
Paper change: The LLM-as-a-judge plausibility evaluation (rubric and prompt) and the full Claude Sonnet replication will be added to the appendix.
We thank the reviewer for noting that AU ties uncertainty to a specific CI parameter, which makes it interpretable, and that AU-min performing worst shows the signal is informative.
Weakness 1 / Question 3: "AU-MAX only seems to be significantly better than Self-ID in 2 out of the 6 configurations in Table 1. However, it requires generation and K forward passes per scenario compared to a singular call for Self-ID. This cost/benefit trade-off should be discussed explicitly."
With five judges, AU-max never loses to Self-ID and wins in 12 of 20 settings, and its extra cost is single-token scoring that also produces the clarification question Self-ID still has to generate.
One may object that Self-ID is enough on ConfAIde+. It is competitive there because one parameter (
data_type) dominates, which a fixed static rule also captures (Table R3); which parameter dominates changes across judges and datasets and can only be found with labels. AU-max needs no such tuning.We agree that Self-ID is a reasonable cheap proxy when the decisive parameter is obvious; AU-max's advantage is largest when ambiguity is spread across parameters.
Paper change: explicit cost/benefit paragraph in Section 5.1.
Weakness 3 / Question 1: "The authors compare a supervised probe to label-free baselines. There needs to be supervised baselines as well."
Supervision explains part of the probe's advantage, but hidden states add error information that no supervised black-box detector recovers.
We added five supervised baselines that use no hidden states: (1) logistic regression on black-box scores (entropy, P(appropriate), self-consistency, Ask4Conf); (2) the same plus AU features; (3) gradient-boosted trees on the features of (2); (4) logistic regression on TF-IDF of the judgment prompt; (5) a single logistic regression on the TF-IDF and the features of (2), concatenated. Every baseline uses the probe's labels and stratified 5-fold splits and is scored out-of-fold.
Table R5. AUROC for detecting initial errors ("Best supervised" = best of the above per row). Columns to be added to Tables 2 and 10.
Paper change: these baselines added to Tables 2 and 10.