- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes PILLAR, which is a PIR-based protocol for privacy-preserving RAG system. In this work, a semi-honest server should learn nothing about the client's query. A formal proof of the security property is provided in the appendix. The authors compare PILLAR against baselines like PACMANN on multiple popular datasets like MS MARCO and SciFact.
Critical strengths
-
The problem being targeted by this work is real and important.
-
The evaluation is thorough.
Critical weaknesses
-
The technical novelty is somewhat limited. Both protocols adapt standard information retrieval structures by fixing the access schedule. It's hard to see the technical challenges to make such adaption.
-
The evaluation, although seemingly thorough, lacks the comparison with more baselines. The major baseline currently in the paper is only PACMANN. The conclusion is hard to generalize without the comparison results against more baselines.
Questions for authors
- Please articulate the technical novelty
3 comments
I appreciate the author's clarification about the novelty. I also checked the replies to comments from other reviewers. I would like to raise my score from 2 to 3.
Thank you! We are glad you see the novelty of PILLAR. Thank you also for reading the comments from the other reviewers, we spent a lot of effort running extra experiments across a variety of configurations to make sure we could clear up any ambiguity about PILLAR’s performance.
Reviewer 8Vw7
We thank the reviewer for recognizing the problem as real and important and the evaluation as thorough. We address both weaknesses and the question in order:
1) Technical Novelty of PILLAR. We thank the reviewer for their feedback on PILLAR’s novelty. We will highlight some novel elements of PILLAR that we may not have made clear in the original text. PILLAR is the first PPRAG to combine both lexical scoring and dense retrieval. The reviewer notes that both PILLAR-Bin and PILLAR-Tree adapt standard information retrieval structures by fixing the access schedule, and that it is hard to see the technical challenge in doing so. The difficulty is fixing the schedule without: (1) leaking information about the client’s query, (2) affecting the quality of the documents retrieved, or (3) increasing the latency. Compared to PACMANN, which requires many rounds of PIR, PILLAR-Bin requires only a single PIR round. As a result, PILLAR-Bin retrieves documents of the same quality faster than PACMANN on every dataset we tested (general comment, Tables 2 and 3). Tiptoe and Coeus also need one round, but only because they scan the entire corpus for every query. PILLAR never scans the entire corpus, and it gives up no quality in exchange: PILLAR-Tree’s best configurations retrieve the highest-quality documents of any private protocol on every dataset (general comment, Table 1).
2) Comparison with more baselines. We agree that comparing PILLAR to PACMANN alone is not enough to generalize our conclusions. We perform multiple new experiments, adding two private baselines (Tiptoe and Coeus), three non-private baselines (BM25, exact dense, and fused hybrid), and three new datasets (FEVER, Natural Questions, and MS MARCO v2). We detail all of these results in the general comment (Tables 1–4). At the same MRR@10, PILLAR-Bin is 2–5× faster than PACMANN, and PILLAR-Tree is 1.2–1.7× faster, on all datasets.
We also address the reviewer’s question:
Q1) Novel achievements of PILLAR. The reviewer asks that we articulate the technical novelty of PILLAR. We articulated the novelty previously for Reviewer LDjq, and we provide the same summary of three novel components of PILLAR: (i) Round-robin interleaving of colliding documents in PILLAR-Bin (ii) a fixed-width beam search over a block-max tree, with the pruning threshold moved to the client in PILLAR-Tree (iii) sparse bound dictionaries in cuckoo tables with self-verifying keys in PILLAR-Tree.