acceptodds
Withdrawn from ICLR 2027

Does Training on a Text Make It More Likely? Not Under Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Likelihood-based data audits assume that a model assigns higher probability to text it was trained on. We test this assumption with a randomized-inclusion design, changing the prediction target while keeping the input prefixes fixed, and find that it fails under distillation. We randomly choose which problems enter training, score every trajectory before and after training, and compare the gain of the included trajectories with the gain of the excluded ones. Fine-tuning on the original tokens gives the included trajectories the larger gain. Using the teacher's argmax labels instead, however, reverses this advantage: the contrast in gains changes from +0.016 to −0.019 nats per token, and the reversal replicates in a second model family. It arises at the roughly 5% of positions where the teacher's label differs from the original token: training only at these positions reproduces the negative contrast, whereas training only at the remaining positions yields a positive one. To first order, the expected contrast is non-negative when training on the scored tokens and can take either sign when the target changes. For auditing, this reversal flips a common membership score, which ranks texts by how much their likelihood rose: the score separates included from excluded trajectories after fine-tuning and ranks them the wrong way round after distillation. A control in which the student is distilled from a copy of itself reverses the score just as much despite a far smaller contrast, so a reversed score alone cannot identify distillation. A randomized contrast in either direction shows that data was used; a null does not show that it was not.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
membership inference, knowledge distillation, on-policy distillation, privacy auditing, memorization, likelihood dynamics, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Q1TROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates whether including a text in training increases its likelihood relative to excluded texts. Using randomized inclusion experiments, the authors show that replacing the original tokens with teacher-provided targets on fixed prefixes can reverse the likelihood-gain contrast between included and excluded trajectories. They also report preliminary on-policy distillation experiments. The paper argues that this reversal can undermine auditing scores that assume training members receive larger likelihood gains.

Critical strengths

None.

Critical weaknesses

1.The connection to practical data auditing is insufficiently established.

In on-policy distillation, the response trajectories are generated by the student itself, whereas an external data owner would typically want to audit whether their prompts or documents were used. The paper does not sufficiently explain how auditing these student-generated trajectories addresses that practical concern, particularly when the data owner cannot access the trajectories generated during training. Although the fixed-prefix experiments establish a controlled counterexample, the practical relevance of this setting needs clearer justification.

2.The additional insight and practical contribution appear limited.

When the supervision target is replaced by teacher labels, increasing the likelihood of the original tokens is no longer the training objective. Consequently, the possibility of a weaker or negative likelihood trace is intuitive. The paper empirically establishes an aggregate reversal, but provides limited predictive understanding of when it occurs and limited evidence of its consequences for auditing methods that calibrate the score direction. Without a stronger demonstration of these consequences or a substantive methodological advance, I find the contribution insufficient for acceptance.

Questions for authors

1.What concrete auditing scenario allows an external data owner to obtain and audit the student-generated trajectories studied in the on-policy experiments? How do the findings translate to auditing externally supplied prompts or documents?

2.Beyond a fixed-direction likelihood score, which auditing methods remain substantially impaired after independent calibration of the score direction?

Reviewer_JJgtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper tests whether training exposure increases a text's relative likelihood when prediction targets are replaced. Randomized inclusion reveals sign reversal under several hard-distillation settings, with implications for likelihood-based audits. The design supports a useful counterexample, although broader conclusions about practical distillation remain less established.

Critical strengths

S1. The randomized-inclusion design addresses an important identification problem, separating the inclusion contrast from broad likelihood drift using matched problems and training settings.

S2. The mechanism is investigated through label-versus-trajectory scoring, disagreement-position interventions, and a first-order analysis, with replication across model families and data pools.

Critical weaknesses

W1. The strongest reversals concern fixed-prefix hard targets from the student or a related teacher; full-vocabulary soft distillation is untested and online evidence remains preliminary.

W2. An almost-identity teacher already produces a reversed audit score, and the online soft effect is close to a numerical floor measured under different precision. Matched-precision and exact-zero-gradient controls would clarify this boundary.

W3. The practical auditing contribution remains limited. Reversing a fixed-direction score is corrected by sign calibration, while the stronger label scores require privileged access. A broader comparison with calibrated audit methods would better establish the operational significance.

Questions for authors

See Weaknesses

Reviewer_P9MSOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper analyses how on-policy distillation (OPD) changes the likelihood of student-generated mathematical solutions. As per the literature, OPD works as follows: for each math question, the student generates solutions, and the teacher is used to assign probabilities to these solutions. The student is trained such that the probabilities that it assigns to these solutions match the ones of the teacher.

The paper does the following analysis: they split a set of math problems into two groups: training and control. They do on-policy distillation on the training set. After distillation, the likelihood of the solutions from the training set is lower than the probabilities of the solutions in the control set. This apparently contradicts the usual setting: fine-tuning on a text makes it more likely.

Critical strengths

Auditing LLMs is very important for understanding their training data, behavior, and methods' strengths.

Critical weaknesses

It is unclear to me what the point of the paper is.

In a usual membership test, we want to test if a fixed text X, let’s say a novel, appears in the training data of a model. Training on a fixed text, via standard next-token prediction, such as usual SFT, makes this text more likely. So usually, low likelihood P(X) suggests training on the text. The point is to detect AI-generated text.

Here, the paper tests something else. Here we analyze the text generated by the model itself. What is the point of this?

All the text that the paper used for comparison is generated by the model in the first place. By definition, on-policy distillation makes the tokens that the teacher and student agree on more likely and the tokens that they disagree on less likely.

The paper uses math questions X and generates solutions Y and then compares P(Y | X) for solutions Y used in OPD vs solutions Y not used for distillation. This might be useful for investigating the behavior of OPD and designing better models, but the question in itself is not useful. It would have been interesting to investigate if the math questions themselves become more likely or not to see if a dataset has leaked into the training data. But this is not done, as the scoring is only done on the solutions Y and not the questions X.

Overall, I think the paper lacks a clear point; thus, I recommend rejection, and I suggest that the authors reconsider the goals and points of the paper.

Questions for authors

I am open to changing my mind if I incorrectly understood the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.