- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces SPriF, an offline framework for comparing GraphRAG evidence releases. It regenerates answers under the same history and uses matched evidence-removal baselines to measure incremental privacy cost from complete outputs and gains in reference-answer likelihood. It then identifies the lowest-cost release satisfying a utility requirement. Experiments on three KGQA benchmarks show that generated answers and prior history affect release comparisons, while higher likelihood-based utility does not guarantee answer correctness.
Critical strengths
- The evaluation method is well designed. SPriF jointly evaluates regenerated answers and evidence under the same history, accounting for privacy effects missed by evidence-only scoring. The experiments show that this can change which release has the lowest cost.
- The experiments are well designed and the results are clearly analyzed. Comparisons with irrelevant history and between learned and mean filling under the same measurement budget help assess each component’s contribution. The paper also distinguishes higher likelihood from correct answers, avoiding overstated claims.
Critical weaknesses
1.The paper does not verify whether lower privacy scores actually make it harder for independent attackers to infer the secret. In the control experiments, the auditor achieves 58.1% accuracy but has an average confidence of 95.4%, showing substantial overconfidence. Comparing results across models is not enough to establish that this score reliably measures actual privacy risk.
Questions for authors
Higher reference-answer likelihood does not guarantee a correct generated answer. Why is it used as the primary utility metric? What additional information does it provide beyond answer correctness, and do the experiments support this choice?
4 comments
Comment 1 (Q1)
Q1: Why use reference-answer likelihood, what does it add beyond correctness, and do the experiments support this choice?
Thank you for this valuable comment. We appreciate your concern and would like to first clarify that reference-answer likelihood is used because it distinguishes releases that receive the same generated-answer correctness label, including releases that both produce an insufficient-evidence response. The reader-value coordinate is the difference in teacher-forced, token-averaged log likelihood of the fixed reference answer between an action’s evidence context and the same case’s matched DROP context, measured in nats per reference-answer token (). AttriBoT (Liu et al., 2025) approximates leave-one-out likelihood effects on a fixed response, while ContextCite (Cohen-Wang et al., 2024) fits a sparse surrogate to response probabilities under context ablations. These methods provide precedents for varying context while holding a response fixed. Here, the fixed target is the reference answer.
Sufficient context need not yield a correct generation (Joren et al., 2025), so we evaluate generated correctness separately. In the 61-case decision cohort, Intermediate alternatives exist in 59/61 cases (96.7%) under the likelihood criterion, but only 26/61 (42.6%) have a Hits@1-correct alternative. Of the cohort’s 2,225 Intermediate actions, 85.0% produce the insufficient-evidence response (). A correctness label cannot distinguish these identical responses within a case, whereas reference-answer likelihood can vary with the evidence context. To address your request for empirical evidence, we have further conducted a post-submission AUROC analysis across 8,206 transformations from 112 cases. We have incorporated these results into of the revised manuscript:
| Correctness label | Pooled AUROC [95% CI] | Case-macro AUROC [95% CI] | Evaluable cases for case-macro |
|---|---|---|---|
| Hits@1 | 0.430 [0.314, 0.584] | 0.768 [0.697, 0.834] | 66/112 |
| Strict exact match | 0.701 [0.451, 0.920] | 0.881 [0.809, 0.940] | 32/112 |
Pooled AUROC uses all transformation rows. Case-macro AUROC equally weights cases containing both positive and negative labels. All four intervals use the same 2,000 whole-case bootstrap resamples from all 112 cases, retaining all actions within each sampled case. Each macro estimate includes only its sampled evaluable cases. Both pooled intervals include 0.5. The case-macro intervals exclude 0.5 on their respective conditional populations. We have not decomposed the difference between these estimates. The labels are the string-based diagnostics defined in , rather than official benchmark evaluation. When correct generated answers are required, the paper reports an explicit correctness-constrained analysis (). Hits@1 does not check whether the released evidence supports the answer. The case-macro results provide evidence of above-chance discrimination averaged over each label’s mixed-label cases. The experiments do not support using reference-answer likelihood as a correctness gate ().
References
- Liu, F., Kandpal, N., and Raffel, C. (2025). AttriBoT: A Bag of Tricks for Efficiently Approximating Leave-One-Out Context Attribution. ICLR.
- Cohen-Wang, B., Shah, H., Georgiev, K., and Madry, A. (2024). ContextCite: Attributing Model Generation to Context. NeurIPS, 37, 95764–95807.
- Joren, H., Zhang, J., Ferng, C.-S., Juan, D.-C., Taly, A., and Rashtchian, C. (2025). Sufficient Context: A New Lens on Retrieval Augmented Generation Systems. ICLR.
Comment 2 (W1)
W1: Does lower measured privacy cost correspond to lower independent recovery?
Thank you for this valuable suggestion. We agree that comparison across models alone is insufficient. We have therefore conducted a recovery evaluation with entailment labels fixed by construction, independently of either model, and incorporated it into of the revised manuscript. LLM inference has been used to recover personal attributes from text (Staab et al., 2024) and to test candidate sanitizations (Pilán et al., 2025). These precedents motivate our use of LLM-based recovery as an empirical check alongside auditor scores. Auditor cost was positive in all 96 entailing views, where Gemma-3-27B-it selected the true value in every case. The synthetic test used a transcript-only five-option format. A history statement binds the target person to a route or medication, and a released key edge gives its value. Both are required to entail the protected fact. The main Qwen auditor’s costs are compared against these construction-defined labels, and Gemma’s recovery is measured separately. Across 48 case-worlds from 24 seeds, mean auditor cost in the 96 entailing views was 3.48 bits. In the 624 non-entailing views, Gemma selected “Cannot be determined from the conversation” in 616, mean cost was 0.07 bits, and 71 exceeded the -bit tolerance.
The original four-option task allowed outside knowledge. Prespecified exclusions retained 35 of 68 matched-history cases. With no conversation shown (subject, relation, and options only), Gemma selected the true value in 28/35 cases. Empty-history DROP also yielded 28/35, with 26 shared successes. At least one action differed from DROP in only 4/35, 2/35, and 1/35 cases under empty, relevant, and irrelevant histories. The relationship between cost rankings and recovery success among transformed actions on the original workload remains unvalidated. The calibration error identified in your review also remains. In the synthetic setting, both auditor cost and recovery distinguish the entailing and non-entailing groups, with the exceptions reported above.
References
- Staab, R., Vero, M., Balunović, M., and Vechev, M. (2024). Beyond Memorization: Violating Privacy Via Inference with Large Language Models. ICLR.
- Pilán, I., Manzanares-Salor, B., Sánchez, D., and Lison, P. (2025). Truthful Text Sanitization Guided by Inference Attacks. Applied Soft Computing, 185, 114013.
A Kind Request for Your Further Consideration
Dear Reviewer EZyM,
Thank you again for your time and thoughtful feedback on our paper. In our response, we clarified the definition and purpose of the likelihood-based reader-value coordinate and reported both pooled and case-macro AUROC results with confidence intervals (). We also added recovery evaluations with a different model family, including a synthetic test with entailment fixed by construction, and explained the remaining limitations of recovery-based validation ().
As the discussion period is drawing to a close, we would greatly appreciate it if you could review our responses and let us know whether they address your concerns. We would be very happy to answer any further questions.
For verification, the supplementary results, frozen inputs, model-output ledgers, and analysis scripts are available under post_submission/ in our anonymous repository. The overview and reproduction instructions are in post_submission/README.md.
Thank you once again for your time and consideration.
Best regards,
The Authors
Overall response to Reviewer EZyM
Thank you for your thoughtful and constructive feedback. We appreciate your positive assessment of the evaluation design and controlled comparisons. We address your comments in two replies: