acceptodds
Under review as a conference paper at ICLR 2027

From Unfamiliar to Familiar: Detecting Pre-training Data via Gradient Deviations in Large Language Models

Abstract

Pre-training data detection for LLMs is essential for addressing copyright concerns and mitigating benchmark contamination. Existing methods mainly focus on the likelihood-based statistical features or heuristic signals before and after fine-tuning, but the former are susceptible to word frequency bias in corpora, and the latter strongly depend on the similarity of fine-tuning data. From an optimization perspective, we observe that during training, samples transition from unfamiliar to familiar in a manner reflected by systematic differences in gradient behavior. Familiar samples exhibit smaller update magnitudes, distinct update locations, and more concentrated updates. Based on this insight, we propose GDS, a method that identifies pre-training data by probing Gradient Deviation Scores of target samples. Specifically, we first represent each sample using gradient profiles that capture the magnitude, location, and concentration of parameter updates across FFN and Attention modules, revealing consistent distinctions between member and non-member data. These features are then fed into a lightweight classifier to perform binary membership inference. Experiments on five public datasets show that GDS achieves state-of-the-art performance with improved cross-dataset transferability over strong baselines. Further analyses reveal distinct gradient distributions between members and non-members, validating our methodology.

Primary area:
generative models
Keywords:
Pre-training Data Detection, Large Language Models, Gradient Deviation, Membership Inference Attack

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8B8cOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Gradient Deviation Scores (GDS), a method designed to detect pretraining data used by large language models. Specifically, GDS uses a lightweight classifier to perform binary membership inference. The authors evaluate the effectiveness of the proposed method on various datasets and models.

Critical strengths
  1. The authors explore potential membership signals derived from sample gradients on target models, which is a valuable attempt at training data detection.
Critical weaknesses
  1. The motivation for the proposed method lacks sufficient justification First, the empirical motivation is established by analyzing parameter updates during LoRA fine-tuning. However, the detection method uses parameter gradients from randomly initialized LoRA adapters. The practical equivalence between these two distinct mechanisms remains unclear. Second, a sound motivation should demonstrate that members and non-members exhibit distinguishable behaviors on the target model. The authors, however, solely analyze member dynamics during fine-tuning. Finally, pre-training data detection is challenging due to the use of single-epoch training. In contrast, the observations presented in the motivational study stem from multi-epoch fine-tuning (e.g., the progressive stabilization of update locations). It remains unclear how these findings apply to pretraining settings.

  2. The manuscript requires substantial improvement in presentation The authors should use standard mathematical notation in the manuscript. For example, “Row_Mean_Std” should be expressed using mathematical notation in the equations. Citation formatting is noticeably inconsistent. Some citations include publication years, while others do not. The authors should carefully revise the captions to ensure they are clear and informative. Overall, the manuscript contains numerous formatting issues and typesetting artifacts, which significantly affect its readability.

  3. The experiments need further strengthening. The distribution shift in the evaluation set may affect the evaluation results in training data detection. The dataset used in “LLM Dataset Inference: Did You Train on My Dataset?” could be a more suitable choice for evaluation.

Suggestions: Please use AI tools thoughtfully and critically. An acceptable paper still requires substantial human effort in its structure, writing, and formatting.

Questions for authors

Questions:

  1. In Figure 1, what metric does the y-axis represent? Why are the x-axis and y-axis labeled identically in the left panel?

  2. As stated in the paper, “For each training sample, we perform inference and backpropagation to obtain LoRA gradients.” For pretrained models, where are the LoRA modules?

  3. The authors analyze parameter updates via LoRA fine-tuning in the motivation experiments, while they use randomly initialized LoRA for the actual detection. Are these two approaches actually equivalent?

  4. Why is the largest change in average update magnitude observed at exposure 0? Exposure 0 indicates that fine-tuning does not occur.

  5. Why are some of the best results highlighted in bold, while others are not? For example, the experimental results in Table 10.

  6. The caption notes that “Bold denotes an obvious difference between exposure 0 and 1.” However, the filled markers in Figures 2c and 2d appear to indicate the minimum values.

  7. Please check the citation formatting. Some citations include publication years, while others do not. This makes the citations appear inconsistent throughout the manuscript.

Reviewer_U8MgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes GDS, a white-box pre-training data detection method. It extracts features from per-example LoRA gradients and trains a small classifier on magnitude, position, and concentration. The method performs well on several membership benchmarks and is also evaluated under cross-dataset transfer.

Critical strengths
  • The gradient features provide useful information beyond likelihood scores. The comparison with the supervised NLL baseline is particularly helpful, since it gives both methods access to membership labels.
  • The experimental results show improvements over competitive baselines across several datasets and model architectures. In particular, GDS outperforms the supervised likelihood-based classifier under the same labeled-data budget, supporting the value of gradient features beyond the benefit of supervision alone.
Critical weaknesses
  • The interpretation of the position features needs more support. Equation 4 measures distance from the matrix center, but it is unclear why this coordinate-dependent quantity should reflect familiarity.
  • Section 3 motivates the three feature categories, but the choice of the eight specific statistics is less clear.
  • The experiments are limited to older models, including GPT-Neo, GPT-J, OPT, Pythia, and LLaMA. It remains unclear whether the observed gradient patterns and detection gains hold for more recent open-weight models with different training data and recipes.
  • Minor typo: "Activate Methods" -> "Activation-Based Methods"
Questions for authors

See above.

Reviewer_zTjBOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Gradient Deviation Scores (GDS) for detecting whether text appeared in an LLM’s pretraining data. It attaches untrained LoRA adapters, performs one forward–backward pass without updating model weights, and summarizes gradients using eight statistics describing magnitude, position, and concentration. A supervised MLP then predicts membership. Across five model architectures, the paper reports strong within-dataset results, including 0.99 AUROC on BookMIA and 0.73–0.76 average AUROC across seven MIMIR domains.

Critical strengths

The proposed GDS extracts information through backpropagation while preserving the original model, avoiding a separate fine-tuning procedure. Also, the author conduct substantial component analysis such as feature, module, rank, initialization, and full-parameter comparisons, which help establish which design choices contribute to performance. Besides, the authors include a supervised likelihood-feature baseline helps distinguish the benefit of gradient features from the benefit of training a classifier with membership labels.

Critical weaknesses
  1. The contribution is insufficiently differentiated from existing gradient-based membership inference methods. OR-MIA [1] already motivates membership detection through optimization behavior and trains a classifier on gradient norms and their responses to perturbations. Thus, the potential distinctive contribution of GDS is therefore the efficient LoRA representation and richer gradient statistics, rather than using gradient behavior for membership inference generally. A comparison against OR-MIA and a simple classifier using per-layer gradient norms, with matched membership-label budgets and reported computational costs, would establish the additional value of the proposed features.
  2. The evaluation does not sufficiently distinguish membership from distributional familiarity. As illustrated in Tab. 4, the removing of timestamps reduces WikiMIA AUROC from 0.96 to 0.84, demonstrating GDS’s sensitivity to temporal cues as removing explicit dates does not eliminate differences in topics, terminology, or writing style, which requires further explanation.
  3. The interpretation of positional features is not sufficiently justified. Eq. 4 measures the distance of large-gradient entries from the matrix center. However, matrix indices do not inherently represent meaningful spatial locations. Under the LoRA factorization, consistently permuting the low-rank coordinates preserves the model’s effective weight update while potentially changing these eccentricity statistics. This is particularly consequential because removing column eccentricity lowers WikiMIA TPR from 0.84 to 0.62, as illustrated in Tab.6. The initialization-seed experiments are useful, but do not resolve this issue. Function-preserving permutation controls would clarify whether these features capture stable membership information or depend on arbitrary coordinate ordering.

[1] Zichen Song, Qixin Zhang, Ming Li, and Yao Shu. “Optimization and Robustness-Informed Membership Inference Attacks for LLMs.” AAAI, 2026 [2] Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi. “Do Membership Inference Attacks Work on Large Language Models?” COLM, 2024.

Questions for authors

Please refer to weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.