acceptodds
Under review as a conference paper at ICLR 2027

Where Membership Lives: Task Abstraction, Exemplar Retrieval, and Privacy Leakage in In-Context Learning

Abstract

In-context learning (ICL) creates a different privacy problem than pretrained and fine-tuned language models: adaptation occurs without parameter updates, while sensitive demonstrations remain explicitly available in the active context and can be re-accessed during inference. We study this contextual membership and ask where membership information resides, how it differs from parameter-updated membership, and how it can be inferred under different model accesses. We introduce controlled behavioral interventions that separate target-specific exemplar information, surface-token sensitivity, and shared task abstraction. A matched ICL–Full-SFT comparison experiment further reveals different privacy footprints: under paraphrased target-associated exposure, ICL yields stronger pooled member- ship signal than SFT (0.869 versus 0.688) and a differently concentrated token-level signal. Finally, replacing private demonstrations with differentially private synthetic demonstrations reduces the evaluated attack to approximately chance, while increasing prediction loss. Together, these results identify active-context exposure as a distinct source of privacy leakage—different from the persistent parameter traces studied in pretraining and fine-tuning—and motivate privacy analyses and defenses specifically designed for ICL.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
in-context learning. membership inference, task abstraction, differential privacy

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_4sMPOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the contextual membership and ask where membership information resides, how it differs from parameter-updated membership, and how it can be inferred under different model accesses. It introduce controlled behavioral interventions that separate target-specific exemplar information, surface-token sensitivity, and shared task abstraction. Additionally, this paper has conducted experiments to analyses the factors and get several conclusions, including ICL yields stronger pooled member-ship signal than SFT (0.869 versus 0.688) and a differently concentrated token-level signal, replacing private demonstrations with differentially private synthetic demonstrations reduces the evaluated attack to approximately chance, while increasing prediction loss.

Critical strengths
  1. This paper mainly analyses the factors affecting the MIA attack in in-context learning and former works have not done it previously.
  2. This paper is technically solid.
  3. This paper provides detailed experiment setups.
Critical weaknesses
  1. This paper is lack of novelty. There are many works including MIA attack in the ICL context and DP synthetic demonstrations generation. And from my point of view, the main contribution is that this paper mainly analysizes three factors in ICL MIA and in-context learning improvement.
  2. This paper only uses one model for the experiments. Since this paper main contribution is some factor discovery, the experiments should be conducted on more models such as llama3.2, qwen3.5 etc. Both thinking and non-thinking mode should be included.
  3. The paper is poorly written. The organization is difficult to follow and from the abstract and introduction, the reader is difficult to get to know the contribution of this paper.
  4. This paper mainly analyses the factors affecting the MIA attack, but there is little difference between GlobalMean and SWM, and thus I think this paper can contribute little to the MIA research community.
Questions for authors
  1. Can you add more experiments on more model architectures inducing qwen3.5, llama3.2 and so on. And please add more experiments on both thinking and non-thinking situation.
Reviewer_6joROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies membership inference and privacy leakage in in-context learning (ICL). The authors focus on how membership information differs from that in fine-tuned models. They examine three factors: exemplar retrieval, token sensitivity, and task abstraction. They analyze token-level membership signals and show that membership inference can achieve near-perfect accuracy using only top-1 token predictions.

Critical strengths
  • The authors perform experiments to examine exemplar retrieval, token sensitivity, and task abstraction to study the sources of membership signals in ICL.

  • The findings that membership signals persist after paraphrasing and that token sensitivity is primarily associated with answer tokens are interesting.

Critical weaknesses
  • The proposed framework primarily relies on controlled prompt interventions to analyze membership signals. Could the authors clarify the technical novelty of this work?

  • The attack assumes that the adversary knows the target sequence and can perform repeated teacher-forced queries. It is unclear how effective the attack would be under more realistic settings, such as when the target sequence is paraphrased in the context, or when the query budget is limited.

  • The SFT baseline is trained for only one epoch, so I wonder whether it has converged. I recommend evaluating additional fine-tuning configurations and comparing membership leakage at similar task utility levels to better support the conclusions about the privacy differences between ICL and SFT.

  • Most experiments use only Qwen2.5-7B, and the main membership inference attack is evaluated on Formal L1 and CNN/DailyMail. I recommend experiments with additional model families and more diverse tasks to show generalizability.

  • I recommend including the following related work on DP prompting for in-context learning in the literature review.

[1] Hong, Junyuan, et al. "DP-OPT: Make Large Language Model Your Privacy-Preserving Prompt Engineer." ICLR, 2024.

[2] Gao, Fengyu, et al. "Data-Adaptive Differentially Private Prompt Synthesis for In-Context Learning." ICLR, 2025.

  • The text in Figures 7 and 8 is too small.
Questions for authors

See Weaknesses.

Reviewer_Eg1fOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates membership inference attacks (MIAs) in in-context learning (ICL). The authors analyze how exemplar retrieval, token sensitivity, and task abstraction contribute to membership information leakage. They also study the distribution of membership signals across token positions and propose a top-1-token-based membership inference method. Finally, the paper explores the use of differentially private synthetic demonstrations to reduce privacy leakage.

Critical strengths
  1. The paper studies privacy leakage in ICL, which is an important security concern for large language models. 2.The proposed top-1-token-based attack achieves strong performance even without access to model probabilities or logits.
Critical weaknesses
  1. Most experiments are conducted using Qwen2.5-7B, with an additional ICL-SFT comparison on Mistral-7B. These models are relatively old, and the experiments lack evaluations on more recent and diverse LLM architectures. Since different models may have significantly different ICL capabilities, it is unclear whether the conclusions generalize to other model families.
  2. ICL performance is highly sensitive to demonstration selection, ordering, formatting, and the number of examples. Although the paper examines different numbers of demonstrations and some controlled interventions, it does not sufficiently investigate how these factors (like order)affect membership inference. This raises concerns about the robustness of the findings under different ICL configurations.
  3. The defense experiments are restricted to a synthetic formal-language task. Although the proposed DP-based approach reduces membership inference performance, it also increases prediction loss substantially. The practical effectiveness and privacy-utility trade-off of this approach remain unclear.
Questions for authors
  1. How well do the findings generalize to more recent LLMs with stronger ICL capabilities, such as newer Qwen, Llama, or other model families?
  2. Since ICL is sensitive to demonstration selection, ordering, and formatting, how would these factors affect the reported membership inference performance? 3-Given that the target example is directly included in the ICL context, how much of the high attack performance comes from simple example copying or matching rather than more general membership-related behavior?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.