- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Existing membership-inference attacks can achieve useful ranking performance while still producing substantially overlapping member and non-member score distributions, leaving an ambiguous decision region. This paper propose OCDR to sovle the above challenge exploiting two complementary Out-RAG statistics to calibrate the membership score.
Critical strengths
- The paper studies an important and timely problem.
- Extensive evaluation covering varying models and datasets.
Critical weaknesses
- My main concern is the motivation of formulating member/non-member score separability as an explicit objective for RAG MIA. What advantages does this offer over existing MIAs which directly determine whether a sample exists in the external retrieval database?
- The threat model seems unrealistic. Although the paper consider a black-box adversary, the reference RAG systems need to be match t he target system in several aspects, including system prompt, number of retrieved documents, the generat or and retriever. These assumptions seem overly strong.
Questions for authors
- The paper consider a black-box access to the target RAG with substantial knowledge, including generator, retriever, number of retrieved documents and system prompt. Is this a practical configuration in real-world RAG deployments? A concrete, realistic instance can strengthen the paper.
- The distinction of the paper from existing calibration MIAs is not clear. Its seems that this work is essentially a calibration, with differences only in its implementation.
- I would consider raising my score if the authors adequately address this concern.