acceptodds
Under review as a conference paper at ICLR 2027

Disambiguating Contextual Integrity Parameters through Uncertainty Quantification

Abstract

Recent work has shown that ambiguity in Contextual Integrity (CI) parameters can substantially impair LLM-based privacy judgments. However, it remains unclear which CI parameter should be clarified and when clarification is worthwhile. We introduce the Contextual Integrity Clarification Framework (CICF), a framework that estimates parameter-level uncertainty by generating candidate specifications of individual CI parameters and measuring their effects on the model's privacy prediction. The framework uses these estimates to select parameters for clarification. We evaluate CICF on the CI benchmarks PrivacyLens+ and ConfAIde+ with three LLM-based privacy judgment models. Our experiments demonstrate that CICF can support more reliable LLM-based privacy judgments through targeted clarification: it consistently identifies more effective clarification targets than random or low-uncertainty baselines, leading to an accuracy improvement of up to 12.0 percentage points in PrivacyLens+ and 6.3 in ConfAIde+ under ideal error detection. We also evaluate a supervised probe for error detection, which combined with our framework still improves accuracy by up to 7.1 percentage points in PrivacyLens+ and 5.7 in ConfAIde+. Sequential adaptive target reselection further improves estimated oracle-gated accuracy beyond one round in all three PrivacyLens+ models, with model-dependent effects on ConfAIde+

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Contextual integrity, privacy reasoning, large language models, uncertainty quantification, ambiguity resolution, clarification, contextual disambiguation, information flow, trustworthy AI

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_3AzJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CICF, a framework for deciding which Contextual Integrity parameter an LLM should clarify when judging whether an information flow is appropriate. For each parameter, it generates candidate specifications and measures aleatoric uncertainty as the mutual information between the model’s judgment and the candidate used. Clarification is then gated by either prediction entropy or a supervised probe on hidden activations that predicts errors. Across three open models on PrivacyLens+ and ConfAIde+, clarifying the highest-uncertainty parameter outperforms random and lowest-uncertainty selection and matches or beats asking the model directly. The authors further show that gains peak at intermediate coverage and that most of the benefit comes from a single clarification round.

Critical strengths

AU ties the model’s uncertainty to a specific CI parameter, which makes it interpretable and useful beyond clarification. Under the oracle gate, AU-MAX beats Random and AU-MIN in all six configurations, and AU-MIN performing worst confirms the signal carries real information about which parameter to clarify.

Critical weaknesses
  1. AU-MAX only seems to be significantly better than Self-ID in 2 out of the 6 configurations in Table 1. However, it requires generation and K forward passes per scenario compared to a singular call for Self-ID. This cost/benefit trade-off should be discussed explicitly.
  2. The proposed method depends heavily on the diversity and quality of the Gemini-generated candidates and on the uniform prior over them. The generator also decides which parameters count as “already specific.” There are no ablations on varying the generator model or validating the quality of the candidates.
  3. The probe needs labeled data for each model and dataset, and it is only evaluated in-distribution (the authors note it is not meant to transfer). While gates beat the provided prior error-detection methods, the authors compare a supervised probe to label-free baselines. There needs to be supervised baselines as well.
Questions for authors

How does the supervised probe for error detection compare with supervised baselines that would have access to the labeled data?

What happens if you change the model that generates the candidates?

How much more expensive is it to run AU-MAX compared to the Self-ID baseline?

4 comments

We thank the reviewer for noting that AU ties uncertainty to a specific CI parameter, which makes it interpretable, and that AU-min performing worst shows the signal is informative.

Weakness 1 / Question 3: "AU-MAX only seems to be significantly better than Self-ID in 2 out of the 6 configurations in Table 1. However, it requires generation and K forward passes per scenario compared to a singular call for Self-ID. This cost/benefit trade-off should be discussed explicitly."

With five judges, AU-max never loses to Self-ID and wins in 12 of 20 settings, and its extra cost is single-token scoring that also produces the clarification question Self-ID still has to generate.

  • Statistics (General Response, Part 2): Wilcoxon signed-rank, Holm per table: AU-max significantly better in 5/10 settings with the paper's candidates and 7/10 label-free (Tables R8a, R8b); Self-ID never significantly better.
  • Why Self-ID loses: in 32% of PrivacyLens+ errors it picks a parameter with AU≈0 (the judge's answer does not change under any candidate); such questions recover only 1–4.5% of errors.
  • Cost: Self-ID = 1 judge call per scenario. AU-max = 5 generator calls (one per CI parameter) + 25 judge calls (5 parameters × K=5 candidates), each returning one token. Scoring 1,824 such prompts takes ≈27 s on one H100 without prefix caching (App. D).
  • Cost reductions: K=2 candidates (10 judge calls) keep 93–100% of the K=5 gain (Table 9), and AU is computed only for scenarios the gate sends to clarification.
  • Not overhead: the candidates are the answer options of the clarification question. Self-ID outputs only a parameter name and still needs a generation step to form the question. AU also gives a magnitude: AU=0 is a stopping rule in multi-round clarification (Table 3); Self-ID is a forced choice.

One may object that Self-ID is enough on ConfAIde+. It is competitive there because one parameter (data_type) dominates, which a fixed static rule also captures (Table R3); which parameter dominates changes across judges and datasets and can only be found with labels. AU-max needs no such tuning.

We agree that Self-ID is a reasonable cheap proxy when the decisive parameter is obvious; AU-max's advantage is largest when ambiguity is spread across parameters.

Paper change: explicit cost/benefit paragraph in Section 5.1.

Weakness 3 / Question 1: "The authors compare a supervised probe to label-free baselines. There needs to be supervised baselines as well."

Supervision explains part of the probe's advantage, but hidden states add error information that no supervised black-box detector recovers.

We added five supervised baselines that use no hidden states: (1) logistic regression on black-box scores (entropy, P(appropriate), self-consistency, Ask4Conf); (2) the same plus AU features; (3) gradient-boosted trees on the features of (2); (4) logistic regression on TF-IDF of the judgment prompt; (5) a single logistic regression on the TF-IDF and the features of (2), concatenated. Every baseline uses the probe's labels and stratified 5-fold splits and is scored out-of-fold.

Table R5. AUROC for detecting initial errors ("Best supervised" = best of the above per row). Columns to be added to Tables 2 and 10.

DatasetModelEntropyBest supervisedProbeProbe − best sup. [95% CI]
PrivacyLens+Qwen3-32B0.6650.7820.866+0.084 [0.055, 0.114]
PrivacyLens+Gemma-40.6010.7800.859+0.079 [0.048, 0.108]
PrivacyLens+Qwen3.6-27B0.6620.7700.834+0.064 [0.034, 0.096]
ConfAIde+Qwen3-32B0.5830.5540.862+0.308 [0.255, 0.359]
ConfAIde+Gemma-40.7060.8420.983+0.141 [0.108, 0.176]
ConfAIde+Qwen3.6-27B0.6940.8030.949+0.146 [0.116, 0.183]
  • We agree that, with labels, black-box detectors beat entropy in 5 of 6 settings, so part of the gap in Table 2 was supervision.
  • Hidden states still add information: the probe beats the best supervised baseline in all 6 settings by 0.06–0.31 AUROC; every CI excludes zero.
  • One may object that a stronger black-box learner would close the gap. The baselines already include gradient-boosted trees on all black-box scores plus AU, and a model that adds prompt TF-IDF, and the smallest gap is still +0.064 [0.034, 0.096].
  • For API-only judges the supervised black-box detector beats entropy and is the fallback; we have not yet evaluated it as a gate.

Paper change: these baselines added to Tables 2 and 10.

Weakness 2 / Question 2: "The proposed method depends heavily on the diversity and quality of the Gemini-generated candidates and on the uniform prior over them. The generator also decides which parameters count as 'already specific.' There are no ablations on varying the generator model or validating the quality of the candidates."

The generator, the label, and the prior change which parameter is picked in individual scenarios, but not the conclusion: AU-max is the best selection rule in every collection and its gain is stable under random priors.

  • Generator. The paper's candidates come from Gemini 3.8 Flash, which sees the label. We built two label-free collections: same generator (Flash) and a different one (Gemini 3.1 Pro). AU-max is the best of four rules (AU-min, Random, Self-ID, AU-max) in all 6 settings with Flash and all 10 with Pro (Tables R6, R2). Without the label, switching generator moves the AU-max gain by <0.8 pp on PrivacyLens+; on ConfAIde+, Pro gives 1.2–2.4 pp more.
  • Uniform prior, explained. To compute AU, each of the K=5 candidates for a parameter needs a probability of being the true specification. The paper assumes all five are equally likely (weight 1/5 each). To test this, we replaced the equal weights with random weights (drawn uniformly over all ways of splitting 1 among 5 candidates, i.e. Dirichlet(1,…,1), new draw per scenario and parameter), recomputed AU, re-selected the parameter, and repeated 200 times. The AU-max gain changed by at most 0.36 pp on average and by <0.9 pp in every draw, in all 20 settings of Tables R1 and R2.
  • Diversity. Candidates are never duplicates and share few words (mean pairwise word Jaccard 0.18–0.25). They are related, not paraphrases: mean embedding cosine 0.63–0.68 (0.32–0.36 for unrelated scenarios); only 0.5–2% of pairs exceed 0.9. In 13–40% of candidate sets the judge's decision differs across the five, the variation AU measures.
  • "Already specific." The share of scenario–parameter pairs declared already specific varies by generator (Flash 33% PrivacyLens+ / 74% ConfAIde+; Pro 15% / 51%), yet AU-max is the best rule under both.

Table R6. Oracle-gate ΔAcc (pp) of AU-max. Cols 1 vs 2: same generator, with/without label. Cols 2 vs 3: no label, two generators.

DatasetModelFlash, label-cond.Flash, label-freePro, label-free
PrivacyLens+Qwen3-32B12.038.829.61
PrivacyLens+Gemma-48.326.156.23
PrivacyLens+Qwen3.6-27B9.136.756.90
ConfAIde+Qwen3-32B4.524.155.33
ConfAIde+Gemma-46.335.117.52
ConfAIde+Qwen3.6-27B3.702.855.04

One may object that the label leak inflates the paper's numbers. It does raise the gain (e.g. 12.03 → 8.82 pp, Qwen3-32B on PrivacyLens+), but the label-free gains remain positive in all settings and AU-max still ranks first, so the ranking of rules does not rely on the label.

The parameter chosen for a single scenario does depend on the candidates (two collections agree on 48–81% of initial errors); AU measures sensitivity to the specifications actually proposed. We did not run a human validation of candidate quality and will list this as a limitation.

Paper change: the label-free Flash collection, the generator and random-prior ablations, and the diversity statistics go in the appendix; the limitation is stated in the paper.

I appreciate the additional ablations. However, candidate quality is not addressed, as there is no metric measuring the plausibility. Additionally, non-Gemini generators are not utilized for the candidates. I will maintain my score.

We thank the reviewer for the follow-up. We address both remaining concerns with new experiments: an LLM-as-a-judge plausibility evaluation for the candidates, and a candidate generator from a different model family.

1. Plausibility metric (LLM-as-a-judge). 83–91% of candidates are judged plausible, and candidate sets contain 4.7–4.9 distinct interpretations out of 5, for both Gemini and non-Gemini candidates.

To evaluate candidate quality systematically, we use an LLM-as-a-judge protocol with two independent judges from different model families (Claude Sonnet 5.5 and GPT-6.1 Terra). The judges evaluate 150 randomly sampled scenario–parameter pairs per benchmark and generator (3,000 candidates total) and never see the ground-truth label.

Each judge rates a candidate as plausible if it meets three criteria:

  • (C0) Compatible: does not contradict any explicit word or constraint in the scenario;
  • (C1) Supported by original wording: is a reading the parameter's original wording can carry, without adding norm-relevant facts (such as consent, authorization, or legal obligations) that may bias the judgment by assumption; and
  • (C2) Confined to the target parameter: leaves every other CI parameter unchanged, so uncertainty is not misattributed.

To measure diversity, each LLM judge groups the 5 candidates of a set into paraphrase clusters; the number of clusters is the number of distinct interpretations (5 = all distinct, 1 = all paraphrases).

Table R9. Candidate plausibility and diversity evaluated via LLM-as-a-judge.

GeneratorBenchmarkPlausible (Sonnet)Plausible (Terra)Plausible (both)Distinct / 5 (Sonnet · Terra)Judge agreement
Gemini 3.8 FlashPrivacyLens+87.3%93.7%86.1%4.69 · 4.8792.0%
Gemini 3.8 FlashConfAIde+86.2%99.1%85.1%4.66 · 4.7986.3%
Claude Sonnet 5.5PrivacyLens+85.9%91.9%83.5%4.86 · 4.9389.8%
Claude Sonnet 5.5ConfAIde+91.1%98.7%90.5%4.89 · 4.9391.6%

2. Non-Gemini generator. AU-max does not depend on Gemini: with Claude Sonnet 5.5 candidates, its gain is higher in all 10 settings. We regenerated every candidate for both benchmarks with Sonnet (identical prompt, label conditioning, and K=5K = 5) and re-ran the full pipeline with the same five judges. AU-max's gain rises by +0.7 to +2.2 pp on PrivacyLens+ and by +1.9 to +12.2 pp on ConfAIde+ (Table R10); the increase is significant in 9/10 settings (paired Wilcoxon, Holm).

Table R10. Oracle-gate ΔAcc\Delta\text{Acc} (pp) of AU-max with label-conditioned candidates from Gemini 3.8 Flash (paper) vs. Claude Sonnet 5.5. Same judges and scenarios.

DatasetModelGemini 3.8 FlashClaude Sonnet 5.5
PrivacyLens+Qwen3-32B12.0313.49
PrivacyLens+Gemma-48.329.01
PrivacyLens+Qwen3.6-27B9.1310.63
PrivacyLens+GPT-6 Luna10.9913.02
PrivacyLens+DeepSeek V4 Flash11.0313.27
ConfAIde+Qwen3-32B4.526.44
ConfAIde+Gemma-46.3312.56
ConfAIde+Qwen3.6-27B3.7010.81
ConfAIde+GPT-6 Luna6.7818.96
ConfAIde+DeepSeek V4 Flash2.197.04

Paper change: The LLM-as-a-judge plausibility evaluation (rubric and prompt) and the full Claude Sonnet replication will be added to the appendix.

Reviewer_HFmmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work analyzes ambiguity in Contextual Integrity privacy judgments by LLMs and investigates which parameter an agent should clarify and when, to help it provide better results. To achieve that they propose the Contextual Integrity Clarification Framework (CICF), which considers both the parameter-level candidate specifications and uncertainty estimation. It uses the aleatoric uncertainty metric to select the parameter to clarify, and an intervention gate to decide when to clarify. This work further expands PrivacyLens+ and ConfAIde+ with 5 candidates per parameter, used as simulated clarifications. The results show that CICF allows better performance than random parameter selection across 3 models (Qwen3-32B, Gemma-4, and Qwen3.6-27B).

Critical strengths
  • The work analyzes the importance of resolving ambiguity in Contextual Integrity by proposing a new framework on Contextual Integrity Clarification Framework.
  • This work discovers that more clarification is not always better, which gives value on why it is important to explore proper weight on the uncertainty.
Critical weaknesses
  • The novelty with respect to CAMBER (Yi et al., 2025) seems limited. CAMBER already studies clarification one CI parameter at a time on the same datasets and identifies which parameters matter the most, while the uncertainty metric follows Hou et al. (2024) and the gates are standard methods. From my understanding, the main new element is selecting the parameter per scenario and deciding when to clarify, but there is no comparison with simple baselines, such as always clarifying the parameter that works best on average. Additionally, AU-max is significantly better than asking the model directly (Self-ID) in only two of six settings, even though Self-ID is much cheaper.
  • It is unclear why clarifying everything hurts. It is counterintuitive, since the candidates are generated to be consistent with the true label and therefore should not push an initially correct judgment in the wrong direction. In Figure 3, the results show that accuracy peaks well below the 100% effective full coverage. The work attributes this to clarifications overturning initially correct judgments, but it never quantifies how often this happens or explains why. From my understanding, this could be partially caused by the selection rule itself, since AU-max picks the parameter whose candidates change the judgment the most, which for a correct scenario is also the one most likely to break it.
Questions for authors
  • How does CICF compare with simple baselines, such as CAMBER that always clarify the parameter that is most effective on average?
  • Does the framework work with stronger closed models, given that the parameter selection and the entropy gate could be evaluated by sampling (Appendix D)? Would AU-max still outperform Self-ID with a stronger judge?
4 comments

We thank the reviewer for recognizing the importance of resolving ambiguity in Contextual Integrity (CI) judgments and the finding that more clarification is not always better.

Weakness 1 / Question 1: "The novelty with respect to CAMBER (Yi et al., 2025) seems limited. [...] there is no comparison with simple baselines, such as always clarifying the parameter that works best on average. Additionally, AU-max is significantly better than asking the model directly (Self-ID) in only two of six settings, even though Self-ID is much cheaper."

CAMBER and CICF answer different questions: CAMBER is an offline, aggregate analysis of how context expansion shifts privacy judgments; CICF is a per-request policy that decides whether to ask and which CI parameter to ask about. Per-request selection beats the strongest static policy the reviewer proposes, even when that policy is tuned with labels.

Evidence (Table R3).

  • Static baseline = always clarify one fixed parameter, chosen after the fact on the full labeled benchmark (favours the baseline).
  • PrivacyLens+: AU-max is significantly better in 9 of 10 settings (+0.8 to +2.2 pp). ConfAIde+: 5 of 10.
  • No static rule is significantly better than AU-max in any of the 20 settings.
  • The best static parameter changes with judge and dataset: data_type for most, recipient for Qwen3-32B on PrivacyLens+, sender for GPT-6 Luna on ConfAIde+ (label-free). AU-max needs no labels to pick it.

One may object that on ConfAIde+ a fixed rule is nearly as good. True, and expected: ambiguity there concentrates in data_type (Table 7), so the per-scenario choice and the fixed one mostly coincide. Where ambiguity is spread (PrivacyLens+, Table 6), the fixed rule loses in 9/10 settings, and which regime holds cannot be known without labels.

Gains over the oracle-tuned static rule are modest (≤ 2.2 pp) and 5/10 on ConfAIde+ are not significant.

Self-ID. With 5 judges and paired tests, AU-max is significantly better in 12 of 20 settings and Self-ID never is (General Response, Part 2).

Table R3. AU-max vs. the best static rule, oracle gate. "Best static rule" = the single parameter that, always clarified, gives the highest ΔAcc for that judge; "Static" = its ΔAcc (pp). Bold: paired-bootstrap 95% CI excludes 0.

DatasetCandidatesModelBest static ruleStaticAU-maxAU-max − static [95% CI]
PrivacyLens+label-cond.Qwen3-32BRecipient+9.90+12.03+2.13 [+0.89, +3.39]
PrivacyLens+label-freeQwen3-32BRecipient+7.44+9.61+2.17 [+1.22, +3.12]
PrivacyLens+label-cond.Gemma-4DataType+6.77+8.32+1.54 [+0.63, +2.49]
PrivacyLens+label-freeGemma-4DataType+4.42+6.23+1.81 [+1.24, +2.37]
PrivacyLens+label-cond.Qwen3.6-27BDataType+9.49+9.13−0.37 [−1.50, +0.83]
PrivacyLens+label-freeQwen3.6-27BDataType+6.06+6.90+0.83 [+0.34, +1.32]
PrivacyLens+label-cond.GPT-6 LunaDataType+9.55+10.99+1.44 [+0.22, +2.66]
PrivacyLens+label-freeGPT-6 LunaDataType+6.92+8.99+2.07 [+1.48, +2.66]
PrivacyLens+label-cond.DeepSeek V4 FlashDataType+9.33+11.03+1.70 [+0.63, +2.86]
PrivacyLens+label-freeDeepSeek V4 FlashDataType+6.86+8.80+1.95 [+1.40, +2.52]
ConfAIde+label-cond.Qwen3-32BDataType+4.67+4.52−0.15 [−0.56, +0.15]
ConfAIde+label-freeQwen3-32BDataType+5.22+5.33+0.11 [−0.56, +0.78]
ConfAIde+label-cond.Gemma-4DataType+5.78+6.33+0.56 [+0.07, +1.15]
ConfAIde+label-freeGemma-4DataType+7.11+7.52+0.41 [−0.22, +1.07]
ConfAIde+label-cond.Qwen3.6-27BDataType+3.56+3.70+0.15 [−0.15, +0.52]
ConfAIde+label-freeQwen3.6-27BDataType+4.70+5.04+0.33 [−0.22, +0.89]
ConfAIde+label-cond.GPT-6 LunaDataType+5.22+6.78+1.56 [+0.70, +2.59]
ConfAIde+label-freeGPT-6 LunaSender+7.78+9.22+1.44 [+0.07, +2.70]
ConfAIde+label-cond.DeepSeek V4 FlashDataType+1.89+2.19+0.30 [+0.07, +0.63]
ConfAIde+label-freeDeepSeek V4 FlashDataType+5.00+6.63+1.63 [+0.89, +2.44]

Paper change: Table R3 and this contrast with CAMBER go into Section 5.1 and the related work.


Weakness 2: "It is unclear why clarifying everything hurts. [...] this could be partially caused by the selection rule itself, since AU-max picks the parameter whose candidates change the judgment the most, which for a correct scenario is also the one most likely to break it."

The reviewer's hypothesis is correct: Table R4 quantifies what Fig. 1 and §3.2 anticipate, that a clarification "may recover an initial error or overturn a correct judgment". It is the reason CICF separates what to clarify (aleatoric uncertainty, AU) from when to clarify (the gate).** AU finds the parameter the judgment is most sensitive to, in either direction; only a gate can keep that sensitivity pointed at wrong judgments.

Evidence (Table R4: every scenario clarified, no gate). Recovery = % of initially wrong judgments that become correct; Break = % of initially correct ones that become wrong.

ModelAU-max RecoveryAU-max BreakAU-max net ΔAccRandom RecoveryRandom Break
Qwen3-32B39.8%13.0%+2.9 pp19.4%4.7%
Gemma-436.0%11.5%−0.5 pp18.4%3.4%
Qwen3.6-27B39.3%11.9%0.0 pp21.6%4.6%
  • AU-max recovers about 2× as many errors as Random and breaks about 3× as many correct judgments. ConfAIde+ shows the same pattern at smaller scale (Break 4.6–5.2% vs. 1.5–1.9%).
  • 70–77% of initial judgments are correct, so breaking ~12% of them costs ~9 pp and cancels most of the recovery: net ≈ 0.
  • Label-consistent candidates can still break a correct judgment: the label constrains the generator, not the judge, which re-decides on a more specific scenario.

One may object that AU is then a poor selector. It is not: its recovery rate is the highest (36–40% vs. 18–22% for Random). The loss comes from applying it to correct judgments, which is what the gate filters; accordingly, accuracy in Figure 3 peaks before full coverage.

Table R4 covers three open judges on PrivacyLens+ (label-conditioned); we do not claim the break rates transfer exactly to other settings.

Paper change: Table R4 and this explanation go into Section 5.2.

Question 2: "Does the framework work with stronger closed models [...]? Would AU-max still outperform Self-ID with a stronger judge?"

Yes for the closed/API judges we added, though they are additional rather than stronger judges.

  • GPT-6 Luna (closed) and DeepSeek V4 Flash return answer-token log-probabilities with reasoning disabled, so AU is computed exactly as in the paper. AU-max is the best selection rule in all 8 new settings (Tables R1, R2) and significantly better than Self-ID in 7 of 8 (Wilcoxon, Holm; Tables R8a, R8b); the eighth also favours AU-max (+0.30 pp).
  • A different, more capable judge does not make Self-ID competitive.
  • They are not more accurate than our open models (60–71% vs. 54–77% before clarification), and we report only the oracle gate for them, since the probe gate needs hidden states the APIs do not expose.

I thank the authors for the detailed response and the new experiments, which clarify why clarifying all scenarios can reduce accuracy and add the comparison with a static baseline that always clarifies the same parameter.

However, Q2 is only partially answered, since the added judges are not stronger than the original models and it is therefore still unclear whether AU-max would outperform Self-ID with a stronger judge. Additionally, the gates were not evaluated for these models.

From my understanding, the static baseline also shows that selecting the parameter per scenario adds only about 1 to 2 pp over always clarifying the same parameter, which suggests that most of the gain comes from clarifying at all. For this reason, the novelty with respect to CAMBER still seems limited to me, and I will keep my score.

We appreciate the reviewer's follow-up, and would like to humbly point out that the novelty concern might rest on a misunderstanding of the static baseline.

1. The static baseline is an oracle. Its parameter is picked with the test labels, separately per judge and dataset, and the pick changes (recipient, sender, data_type). No deployable system can run it. Label-free AU-max still beats it in 18/20 settings, significantly in 14/20, and is never significantly worse.

2. The gain does not come from "clarifying at all". Random also clarifies one parameter per scenario and gains +4.3 to +6.0 pp on PrivacyLens+; AU-max gains +8.3 to +12.0 pp (Table R1). Choosing the parameter roughly doubles the gain.

3. Novelty vs. CAMBER. CAMBER's gains come from expansions generated with the gold label (its Sec. 5.1–5.2): an LLM, given the label, picks a field and writes context pushing the example toward that label; which field was picked is only analyzed afterwards. Its own limitations (Sec. 7) state that it "does not provide a solution for obtaining user clarifications in practice." CICF chooses what and whether to ask without labels, and its gains hold with label-free candidates (Table R2).

4. Q2. We tried a stronger judge, GPT-6.1 Terra (label-free candidates, oracle gate), which is more accurate than every original judge before clarification (80.8% vs. ≤76.9% on PrivacyLens+; 79.1% vs. ≤68.1% on ConfAIde+). AU-max still achieves statistically significant gains over Self-ID on both datasets. This adds to GPT-6 Luna and DeepSeek V4 Flash, where AU-max beats Self-ID significantly in 7/8 settings. The probe gate needs hidden states, which these APIs do not expose; we state this limitation.

Reviewer_FHZUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CICF, which uses uncertainty to decide what privacy-related information an LLM should clarify and when. Experiments show improvements over random selection and entropy-based methods, but rely on generated candidates rather than real user clarifications.

Critical strengths
  1. CICF identifies which CI parameter most affects the judgment and outperforms random selection.
  2. The paper separates what to clarify from when to ask, showing that unnecessary clarification can hurt accuracy.
Critical weaknesses
  1. Candidate generation uses ground-truth labels, potentially overstating the gains.
  2. Sequential clarification relies on an oracle error gate, while the Probe requires labeled data and white-box access.
Questions for authors
  1. Does CICF still improve accuracy when candidate generation has no access to ground-truth labels?
  2. Does CICF work when users provide their own clarifications instead of generated candidates?
  3. Given the extra forward passes, why use AU-Max when Self-ID achieves similar results in most settings?
  4. Can the Probe generalize to unseen scenarios or other models without retraining?
  5. Do the gains persist with a practical intervention gate, and are they worth the extra interaction cost?
2 comments

We thank the reviewer for detailed feedback and would like to respond as follows.

Weakness 1 / Question 1: "Candidate generation uses ground-truth labels, potentially overstating the gains." "Does CICF still improve accuracy when candidate generation has no access to ground-truth labels?"

Label access inflates the size of the gain, but not the conclusion. Without labels, AU-max is even better selection rule in every setting.

  • We regenerated all candidates without the label (Gemini 3.1 Pro) and re-evaluated with 5 judges. AU-max is the best rule in all 10 model–dataset settings, with gains of +5.0 to +9.6 pp (Table R2, General Response).
  • Without labels, AU-max beats Self-ID (the model naming the parameter it is unsure about) in 7 of 10 settings (Wilcoxon signed-rank, Holm-corrected). Self-ID is never significantly better (Table R8b).
  • One may object that the remaining gain is still an artifact of the generator. With the same generator, removing the label lowers the AU-max gain by 2.2–3.2 pp on PrivacyLens+ and 0.4–1.2 pp on ConfAIde+, and the ranking of rules stays the same (Table R6, response to 3AzJ).
  • We agree that labels inflate the gains, and that the label-free numbers are the ones to trust. They are also pessimistic: a label-free candidate can describe a context where the benchmark label no longer holds, and we still score the judge against that label.

Paper change: we will add the label-free results (Table R2) next to the label-conditioned ones in RQ1.

Weakness 2 / Question 5: "Sequential clarification relies on an oracle error gate, while the Probe requires labeled data and white-box access."

In RQ3, the oracle gate is a control that isolates what to ask in a second round. When to ask is a separate question, answered by the practical gates of RQ2. Neither result depends on the other.

  • Why an oracle in RQ3. RQ3 compares Repeat (clarify the same parameter again) with Adaptive (clarify the new AU-max parameter) on judgments that start out wrong, which are the only ones a clarification can fix. An imperfect gate would mix this comparison with the gate's false positives. As the paper states (§6), the oracle gate "isolates the comparison between alternatives" but "does not establish the performance of a deployable (imperfect) intervention policy".
  • The comparison is informative (Table 3). Repeat changes ΔAcc by less than 0.2 pp because the generator judges the parameter already clear in ≥96% of updated scenarios; Adaptive selects a different parameter in ≥96% of updated PrivacyLens+ scenarios.
  • Practical gates, RQ2. Here the gate does not know which judgments are wrong: it clarifies the q% of scenarios with the highest predicted error. With the probe as gate, CICF improves accuracy by up to 7.1 pp on PrivacyLens+ and 5.7 pp on ConfAIde+ (Figure 3; e.g. Qwen3-32B on PrivacyLens+ goes from 69.8% to about 76.9% with 45% of scenarios clarified).
  • One may object that the probe just moves the label requirement to the gate. The probe (a linear classifier on hidden states that predicts initial errors) needs only 50 labeled scenarios to beat the best label-free detector in all 6 settings (AUROC 0.70–0.92 vs. 0.58–0.71; Table R7 in Part 2, App. C).
  • White-box access is optional; it buys accuracy, not feasibility. (i) A probe trained on outputs only (same labels, no hidden states) reaches AUROC 0.55–0.84, against 0.83–0.98 for the white-box probe (Table R5, response to 3AzJ): lower in all 6 settings, by 0.06–0.31, so hidden states improve error detection but are not required. (ii) Baseline entropy needs neither labels nor training; it is weaker still (0.58–0.71, below the output-only probe in 5 of 6 settings) but still carries error signal, and it is the label-free practical gate in RQ2.
  • We do not report multi-round gains under a practical gate. The Fig. 4 numbers are an upper bound for what a second round can add.

Paper change: Section 5.3 will state that the multi-round gains in Fig. 4 assume the oracle gate and are an upper bound.

Question 2: "Does CICF work when users provide their own clarifications instead of generated candidates?"

While a complete user study will be conducted in future work, our new label-free setting (described above) provides a close proxy of the practical implications, and there the ranking of selection rules hold.

  • Label-free candidates (Table R2) are any plausible specification, not steered toward the correct judgment, and AU-max is still the best rule in all 10 settings.
  • We have no real-user study. Real answers can be partial or fall outside the candidate set. The paper states this limitation, and it is our main direction for future work.

Question 3: "Given the extra forward passes, why use AU-Max when Self-ID achieves similar results in most settings?"

Self-ID matches AU-max only when one parameter obviously dominates, and that cannot be known without labels. Without labels, AU-max is significantly better in 7 of 10 settings and never worse (Table R8b). The cost is single-token scoring, reduced further by K=2 candidates and the gate (Table 9). Full argument in General Response, Part 2.

Question 4: "Can the Probe generalize to unseen scenarios or other models without retraining?"

It generalizes to unseen scenarios, only partly to an unseen benchmark, and not to other models. Retraining is cheap: 50 labels are enough to beat every label-free detector.

  • Unseen scenarios: yes. All probe results are out-of-fold (stratified 5-fold), so no scenario is scored by a probe that saw it.
  • Unseen benchmark: partly. A probe transferred unchanged from the other benchmark reaches AUROC 0.71–0.83, against 0.83–0.98 when trained on the target benchmark. With 50 target labels it reaches 0.70–0.92, above the label-free detector in all 6 settings (Table R7).
  • Other models: no (concession). The probe reads one model's hidden states, so each model needs its own probe at the 50-label cost above.

Paper change: Table R7 and these limits go into App. C.

Table R7. Extends the label-efficiency analysis of App. C. AUROC for detecting the model's initial errors on the test benchmark.

  • No labels: best detector that needs no labeled data (Tables 2 and 10 of the paper).
  • Transferred: probe trained on the other benchmark, applied unchanged.
  • 50 labels: probe trained on 50 labeled scenarios of the test benchmark, evaluated on a held-out 20% (mean of 10 random draws).
  • All labels: probe trained on the test benchmark (5-fold, out-of-fold), as in the paper.
ModelTest benchmarkNo labelsTransferred50 labelsAll labels
Qwen3-32BConfAIde+0.5830.7090.7020.862
Qwen3-32BPrivacyLens+0.6650.7060.7290.866
Gemma-4ConfAIde+0.7060.8310.9200.983
Gemma-4PrivacyLens+0.6010.7850.7630.859
Qwen3.6-27BConfAIde+0.6940.7840.8730.949
Qwen3.6-27BPrivacyLens+0.6620.7190.7170.834

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.