acceptodds
Under review as a conference paper at ICLR 2027

Dataset Inference via Cumulative Evidence

Abstract

Dataset Inference (DI) aims to determine whether a given dataset was used to train a model. However, existing DI methods for large language models (LLMs) face two significant practical limitations. First, they assume gray-box access to token-level probabilities, whereas most commercial LLMs expose only generated text through black-box APIs. Second, they rely on rigid p-value hypothesis testing, which requires committing in advance to a specific dataset size and significance level. In practice, this makes auditing either too costly or statistically inconclusive, since adaptively adding additional samples invalidates standard p-value guarantees. We address these limitations with a fully black-box and sequential approach to DI. To overcome API restrictions, we introduce a novel method for estimating continuous per-token probabilities directly from discrete, label-only outputs, unlocking **DI in a fully black-box setting**. To address the rigidity of standard fixed-sample testing, we identify that e-values are particularly well suited for this setting. Based on e-values, we create a new sequential testing framework for DI. Crucially, when the initial dataset size is insufficient, our framework seamlessly accommodates additional data collection without violating statistical guarantees, enabling **DI with iterative evidence accumulation**. Together, these advances make DI practical for real-world LLM auditing. We validate our novel DI method on 6 diverse open-source LLMs with known training datasets and demonstrate its efficiency in production settings on 6 LLMs exposed via public APIs. Our cost analysis shows that, for instance, auditing GPT-5 via the OpenAI API using the WikiMIA-24 dataset with our method costs only $1.92.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
llm, dataset inference, membership inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_et79OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes DICE, a black-box dataset-inference framework that combines text-derived membership features with sequential testing by betting. Token similarities are transformed into proxy probabilities, and evidence is accumulated using a kernel-based discrepancy and adaptive betting strategies.

Critical strengths

The text-only access setting is practically relevant, and sequential auditing is a sensible objective when model queries are costly.

Critical weaknesses
  1. The OLMo-7B/Wikipedia results are inconsistent between Tables 1 and 8.
  2. Sigmoid-transformed cosine similarities lie approximately in [0.269, 0.731] and do not identify underlying token probabilities. Please distinguish proxy-feature utility from probability recovery and compare alternative mappings under identical features, preprocessing, and tests.
  3. Rejecting feature-distribution equality does not establish training membership without controlling unrelated distribution shifts, particularly with temporal proxy labels.
  4. Improve the clarity of the figures and standardize their formatting.
Questions for authors
  1. How do the authors explain the elevated FPRs, including 0.087 for DICE on Pythia-12B/OpenWebText2 (Table 3) and 0.644 for PETAL on Pythia-410M/PhilPapers (Table 5)?
  2. Can the authors establish anytime-validity for the complete preprocessing-and-betting procedure, particularly the data-dependent trimming?
  3. What is Table 2 intended to show?
1 comment
AuthorsOpenReview ↗

Response to Reviewer et79

Thank you for identifying issues in the interpretation of the proxy scores, the validity argument, and the consistency of the reported results.

1. Inconsistent OLMo-7B/Wikipedia results

We agree that Tables 1 and 8 are inconsistent. We will trace both entries to their run configurations and aggregation records, rerun the setting if those records cannot resolve the discrepancy, and regenerate the affected tables from one verified result source. We will also check the remaining model/dataset labels and summary statistics.

2. Sigmoid similarities and probability recovery

You are correct that applying a sigmoid to cosine similarity yields values approximately in [0.269, 0.731] and does not recover the model's token probabilities. Its intended role is a bounded similarity-derived feature. We will correct the probability-recovery language.

We will compare sigmoid, a positive bounded cosine mapping, PETAL calibration, and exact probabilities under matched feature definitions, preprocessing, sequential tests, and tuning budgets. We will document the domain and clipping rules for logarithmic features and evaluate downstream audit utility.

3. Distributional differences versus membership

The formal null concerns equality of feature distributions. Rejecting it supports membership attribution only when unrelated temporal and domain differences are adequately controlled.

We will add model-free baselines, genuinely unexposed temporal/domain controls, and a randomized controlled-exposure experiment. Qwen and commercial-model results will be identified as proxy-label case studies, distinct from evaluations with documented training inclusion.

4. Figure clarity and formatting

We will standardize fonts, colors, legends, model names, axes, and significance thresholds. Captions will distinguish individual evidence trajectories, averages, and rejection probabilities and define the displayed uncertainty. We will also replace the non-rejection label in Figure 1 with “inconclusive within the available budget.”

5. Elevated FPRs for DICE and PETAL

We will investigate both DICE's 0.087 and PETAL's 0.644, including the shared testing backend and method-specific feature generation, calibration, and preprocessing. The checks will compare the original procedure with unprocessed features, independently calibrated preprocessing, and strictly past-based shared preprocessing. We will repeat null trials, report rejection counts and uncertainty intervals, and rerun affected comparisons. Weak features alone do not explain inflated false rejection under a valid null procedure.

6. Anytime validity of preprocessing and trimming

The trimming description leaves unclear whether current observations influence the thresholds or replacement values. We will make these dependencies explicit and ensure that the common transformation, witness, and stake are fixed before observing each new pair.

Under conditional pair exchangeability, swapping the pair reverses the bounded discrepancy score while leaving those choices unchanged, giving zero conditional mean. With admissible stakes, the resulting nonnegative wealth process supports the anytime error bound. We will verify these conditions for the complete implementation, including swap symmetry and independence from future data, and rerun affected experiments.

7. Purpose of Table 2

Table 2 qualitatively illustrates the continuation interface using the same prefix and outputs from different APIs. It does not establish training membership; a plausible continuation can arise through generalization. We will clarify its caption and role in Appendix A.6. The planned API analysis will report systematic continuation-compliance measurements.

Reviewer_QYT2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies dataset inference for LLMs, which checks whether a whole dataset was used to train a model. The existing methods have two problems. They need token probabilities, which most commercial APIs do not give, and they use p-value tests where the sample size must be fixed before testing. The paper proposed, DICE, and it has two parts. First, it builds membership features from generated text only. The model continues a prefix, and the output is compared with the real text using embedding similarity, at the sequence level and the token level. Second, it uses a sequential test based on e-values (testing by betting with kernel MMD), so the auditor can keep adding samples and stop once the evidence is strong enough. Experiments on Pythia with the Pile show that DICE reaches significance with fewer samples than other black-box baselines.

Critical strengths
  • It is intuitive to use a sequential test with e-values. The auditor can start with a few samples, add more if the result is unclear, and stop early when the evidence is enough.
  • The method is simple. It does not need a surrogate or shadow model, unlike PETAL, and all features come from two types of queries.
  • The evaluation performance on Pythia model using PILE dataset is a practical choice, since PILE is a ground-truth source that Pythia models used to train.
  • There are many ablations: betting strategy, betting cap, embedding model, normalization and trimming, and a polluted held-out set.
Critical weaknesses
  • The test checks whether suspect and held-out features have the same distribution. However, WikiMIA-24 and BookMIA split by date, so a rejection can come from temporal shift alone, which [1] showed that this shift is what makes membership inference appear successful on such benchmarks. The paper’s false-positive experiments use only IID splits, so they do not rule this out.

  • Qwen2’s training data is unpublished and BookMIA’s labels were built for OpenAI models. The paper still calls the data “known” and concludes membership from its own rejection.

  • The paragraph at line 216 is cut off. It does not give any ideas on how the paper transforms the features into black box features.

  • The paper claims that the experiments are done on “6 open-source LLMs”, but Qwen results are missing from the main experimental results its results are only in Appendix D.2, without any baselines.

  • The paper claims that the GPT-5's weak performance on WikiMIA-24 (cut-off date is March 2024) is due to the cut-off date of GPT-5 is September 2024, it will have some contaminations in the training dataset of GPT-5. However, this is not convincing, GPT-4.1 has a cut-off date as June 2024 and same as Gemini 2.0 flash (the cut-off date is also June 2024). Both cutoffs are after March 2024, so their held-out sets should be contaminated too. But Gemini is the fastest model in the table (38 samples) and GPT-4.1 reaches TPR 1.0. If a late cutoff caused the slowdown, these two should also be slower, and they aren’t.

  • In Figure 2, DICE accumulates evidence faster than DICE+ on Pythia-12B and Pythia-6.9B. Table 6 shows the same pattern on several subsets, for example FreeLaw (TPR 0.790 vs 0.059) and GitHub (0.697 vs 0.011). Since DICE+ uses the exact token probabilities from the model rather than a proxy, I would expect it to be an upper bound for DICE. Could the authors explain why it performs worse?

  • In the paper, the main claim that the paper states that the test is run at the 5% level, so FPR should stay at or below 0.05. For DICE the tables show:

    • OpenWebText2 on Pythia-12B: 0.087 (Table 3)
    • OpenWebText2 on Pythia-6.9B: 0.059 (Table 4)
    • GitHub on Pythia-410M: 0.060 (Table 5)

    The paper should explain these entries or resolve the overclaiming of the paper on this.

[1] LLM Dataset Inference: Did you train on my dataset?

Questions for authors
  1. Can the authors show that the rejections on WikiMIA-24 and BookMIA come from membership and not from the temporal shift between the suspect and held-out sets? For example, a control with a model whose cutoff is before both sets, or a model-free baseline on the same split.
  2. What evidence shows that BookMIA is in Qwen2's training data, given that the corpus is unpublished and the labels were built for OpenAI models? Which cutoff was used to set the member and non-member labels for Qwen2?
  3. Qwen2 is counted among the "6 open-source LLMs", but its results are only in Appendix D.2 without baselines. Could the authors add baseline comparisons for Qwen2, or adjust the claim?
  4. If the later cutoff explains GPT-5's weaker result, why are GPT-4.1 and Gemini 2.0 Flash, whose cutoffs are also after March 2024, not affected? How many held-out samples fall before each model's cutoff?
  5. Why does DICE+, which uses exact token probabilities, perform worse than DICE in Figure 2 and on FreeLaw and GitHub in Table 6? Was DICE+ tuned separately, or does it reuse the hyperparameters chosen for DICE?
  6. Why does the FPR exceed 0.05 for OpenWebText2 (Tables 3 and 4) and GitHub (Table 5), given the claim of strict Type-I error control?
2 comments
AuthorsOpenReview ↗

Response to Reviewer QYT2

Thank you for the concrete questions about label provenance, temporal confounding, comparison fairness, and result consistency. We address each below.

1. Temporal shift versus training membership

We agree that rejection on a date-based split does not isolate training membership, and IID null controls do not resolve this concern.

We will add model-free content baselines on the same splits, temporal negative controls in which both groups verifiably postdate frozen model weights, and source/topic/length-matched comparisons. We will also run a controlled-exposure experiment that randomizes document inclusion within time/domain strata. Splits will respect document or book boundaries and exclude duplicates. These controls will be reported separately from proxy-label case studies.

2. Qwen2 membership labels and cutoff

We do not have independently verified sample-level evidence that BookMIA was included in Qwen2's training corpus. Labels designed for OpenAI models and rejection by DICE do not establish that membership.

We will check the original labeling configuration and report its source, any model-specific relabeling, and the cutoff actually used. We will present Qwen results as proxy-label case studies and remove the claim that their training membership is known.

3. Incomplete paragraph at line 216

The paragraph is incomplete, and we apologize for the omission. At each position, DICE queries the next token using the original text prefix, computes the embedding cosine similarity between the predicted and true tokens, and applies a sigmoid to obtain a bounded proxy score. These scores are then used to construct the token-level features. We will restore this explanation and the feature definitions in the main text, using proxy-score terminology.

4. Six-model claim and missing Qwen baselines

The six open-weight models comprise three Pythia models, OLMo-7B, and two Qwen2 variants. Open weights do not imply known training membership. We will revise the model inventory to distinguish the Pythia/OLMo evaluations from the Qwen proxy-label studies. We will add matched baseline comparisons for both Qwen variants and summarize their results in the main text.

5. GPT-5 performance and knowledge cutoffs

Your comparison with GPT-4.1 and Gemini is valid: a later reported cutoff alone does not explain GPT-5's slower evidence accumulation. The current evidence does not establish the cause, and we will remove this causal attribution.

We will report the tested model snapshots, date definitions, and the number and proportion of held-out texts preceding each model's reported cutoff. These counts indicate possible exposure, not verified contamination. We will examine prompting and decoding effects separately through controlled comparisons.

6. DICE+ performance and tuning

DICE+ is a particular pipeline using exact-probability statistics, not a formal upper bound over all text-derived features and decision rules. Exact probabilities do not guarantee higher power after a fixed feature transformation and test. This does not explain the observed FreeLaw and GitHub gaps.

The manuscript does not document separate tuning. We will verify the original configurations and compare both methods with matched feature scope, preprocessing, and testing, using equal independent tuning budgets on separate development data. Stable log-probability processing and joint probability/proxy features will help examine the source of the gaps.

7. FPR above the nominal 0.05 level

The reported values of 0.087, 0.059, and 0.060 require explicit investigation. We will reproduce these settings and audit null-set construction, preprocessing, witness/stake update order, and result aggregation. We will report rejection counts, denominators, and uncertainty intervals, distinguishing independently randomized null splits from repeated orderings of the same split. Any necessary changes will be followed by reruns of all affected results. The cause remains unresolved pending these checks.

Thank you for the detailed response. The explanation of the token-level features clears up my question about line 216, and I agree that using exact probabilities does not guarantee that DICE+ will outperform DICE. I also appreciate your plans to clarify the status of the Qwen2 labels and remove the cutoff-based explanation for GPT-5. My main concern is still how to interpret a rejection on WikiMIA-24 or BookMIA, since differences between older and newer texts could produce the same result. If you have results from the proposed control where both groups postdate the model’s frozen weights, I’d be interested to see them. I also recognize that an empirical FPR can exceed 0.05 by chance. For the 0.087 reported on OpenWebText2 with Pythia-12B, could you clarify how many independent null splits contributed to that estimate and how much uncertainty there is once repeated orderings of the same data are taken into account? These two points remain open for me, so I’m keeping my rating at 2.

Reviewer_exhnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DICE, a black-box dataset inference pipeline that compares a suspect dataset with a held-out dataset. It extracts sequence-level continuation similarities and token-level semantic similarities, maps the latter through a sigmoid to construct probability-like proxies, and derives multiple membership features. These feature vectors are compared using an existing sequential kernel-MMD testing-by-betting framework, with ONS or approximate Kelly betting and a wealth-based rejection threshold. The paper evaluates the pipeline on Pythia, OLMo, and Qwen models, and on commercial APIs using WikiMIA-24. Its principal claims concern black-box applicability, anytime-valid error control, and reduced auditing cost.

Critical strengths
  1. The studied problem is important.
  2. The experiments span several model families and text domains, and include feature and preprocessing ablations.
Critical weaknesses
  1. The statistical testing component is adopted from prior work: Sequential testing by betting, the MMD witness construction, and the betting strategies are established methods that the paper explicitly adopts. The manuscript does not clearly identify a new statistical contribution beyond their application to dataset inference.

  2. Limited novelty in the overall method: The pipeline largely combines existing black-box membership signals and feature constructions with an established sequential test. It does not clearly demonstrate a new technical mechanism or a substantive challenge resolved by this integration.

  3. Insufficient justification of the specific testing design: The paper motivates adaptive auditing in general, but does not adequately justify why its particular betting-based MMD construction is especially suitable for dataset inference. Its advantages over alternative valid sequential tests and its compatibility with the characteristics of the extracted signals remain insufficiently established.

  4. Limited validation of broad black-box applicability: The evaluation does not systematically isolate how post-training, prompting choices, and decoding or API constraints affect inference. Consequently, the broad claims of applicability to modern commercial LLMs extend beyond the conditions actually validated.

  5. Insufficient analysis of failure modes: Although unsuccessful results are reported, the paper lacks a systematic analysis of why detection fails across domains and model settings. The operating conditions under which an audit is reliable, inconclusive, or vulnerable to misleading evidence are therefore insufficiently characterized.

  6. Incomplete theoretical support for the full pipeline: The analysis mainly invokes existing validity results without fully establishing that the complete preprocessing and adaptive testing pipeline satisfies their assumptions. These results also do not by themselves establish the claimed power, exponential evidence growth, or sample-efficiency advantages.

Questions for authors

Please see weaknesses above.

1 comment
AuthorsOpenReview ↗

Response to Reviewer exhn

Thank you for recognizing the importance of text-only auditing and for identifying where the contribution, design choices, and guarantees need clarification.

1. Statistical novelty

We agree that e-values, the MMD witness, and ONS/Kelly betting are established methods. Our contribution is their application within a text-only dataset-auditing procedure, together with the construction of its audit features. We will make this scope explicit and revise the attribution to prior statistical work. The full-pipeline validity analysis should establish the correctness of this application, not be presented as a new testing principle.

2. Contribution of the overall method

DICE addresses two practical requirements together: obtaining audit signals when target-model probabilities are unavailable, and acquiring additional samples when the required audit budget is unknown. It combines sequence-level continuation similarity with token-level similarity-derived features and retains their multivariate structure during testing. The feature construction adapts probability-based membership statistics without fitting a surrogate probability calibrator. This task-specific design and its evaluation constitute the intended contribution; the individual statistical tools build on prior work.

3. Why betting-based MMD?

MMD accepts multivariate features, and its witness can be estimated from past observations for sequential evidence accumulation. These properties motivate our choice but do not establish superiority over other valid sequential tests.

We will compare it with an alpha-spending MMD/permutation test and an exponential-mixture test on the same bounded payoff stream. We will match features, error levels, tuning budgets, and query budgets, and report false-positive rates, power, stopping costs, and computation. Costs will include both data streams and calibration; non-rejecting runs will be charged their full budget.

4. Post-training, prompting, decoding, and API constraints

We will add matched comparisons of base and instruction-tuned models and vary prompts, decoding settings, and output-length constraints while holding audit data and budgets fixed. We will report continuation compliance, refusals or failed queries, feature changes, and audit performance, and document controls unavailable through each API. Applicability claims will be limited to the evaluated access conditions.

5. Failure modes and operating conditions

We will analyze successful and weak-detection settings, including ArXiv and OpenSubtitles, through feature separation, continuation compliance, per-run evidence trajectories, feature-block ablations, and kernel comparisons. Matched-reference and temporal/domain controls will help distinguish weak signals, interface effects, limited budgets, and unrelated distribution shifts. We will describe failure to cross the threshold as inconclusive within the available budget.

6. Full-pipeline validity and power claims

Our validity argument assumes conditional exchangeability of the next pair under the null and preprocessing, witness, and betting choices determined before that pair is observed. The current trimming description does not sufficiently establish these conditions.

We will specify and audit the complete update order, give the validity argument for a common preprocessing transformation applied to both streams, and rerun affected null experiments. We will separate this conditional error guarantee from power and efficiency claims. The degree-two polynomial kernel does not distinguish every distributional alternative, and validity alone does not guarantee exponential evidence growth or sample savings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.