- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes DICE, a black-box dataset-inference framework that combines text-derived membership features with sequential testing by betting. Token similarities are transformed into proxy probabilities, and evidence is accumulated using a kernel-based discrepancy and adaptive betting strategies.
Critical strengths
The text-only access setting is practically relevant, and sequential auditing is a sensible objective when model queries are costly.
Critical weaknesses
- The OLMo-7B/Wikipedia results are inconsistent between Tables 1 and 8.
- Sigmoid-transformed cosine similarities lie approximately in [0.269, 0.731] and do not identify underlying token probabilities. Please distinguish proxy-feature utility from probability recovery and compare alternative mappings under identical features, preprocessing, and tests.
- Rejecting feature-distribution equality does not establish training membership without controlling unrelated distribution shifts, particularly with temporal proxy labels.
- Improve the clarity of the figures and standardize their formatting.
Questions for authors
- How do the authors explain the elevated FPRs, including 0.087 for DICE on Pythia-12B/OpenWebText2 (Table 3) and 0.644 for PETAL on Pythia-410M/PhilPapers (Table 5)?
- Can the authors establish anytime-validity for the complete preprocessing-and-betting procedure, particularly the data-dependent trimming?
- What is Table 2 intended to show?
Response to Reviewer et79
Thank you for identifying issues in the interpretation of the proxy scores, the validity argument, and the consistency of the reported results.
1. Inconsistent OLMo-7B/Wikipedia results
We agree that Tables 1 and 8 are inconsistent. We will trace both entries to their run configurations and aggregation records, rerun the setting if those records cannot resolve the discrepancy, and regenerate the affected tables from one verified result source. We will also check the remaining model/dataset labels and summary statistics.
2. Sigmoid similarities and probability recovery
You are correct that applying a sigmoid to cosine similarity yields values approximately in [0.269, 0.731] and does not recover the model's token probabilities. Its intended role is a bounded similarity-derived feature. We will correct the probability-recovery language.
We will compare sigmoid, a positive bounded cosine mapping, PETAL calibration, and exact probabilities under matched feature definitions, preprocessing, sequential tests, and tuning budgets. We will document the domain and clipping rules for logarithmic features and evaluate downstream audit utility.
3. Distributional differences versus membership
The formal null concerns equality of feature distributions. Rejecting it supports membership attribution only when unrelated temporal and domain differences are adequately controlled.
We will add model-free baselines, genuinely unexposed temporal/domain controls, and a randomized controlled-exposure experiment. Qwen and commercial-model results will be identified as proxy-label case studies, distinct from evaluations with documented training inclusion.
4. Figure clarity and formatting
We will standardize fonts, colors, legends, model names, axes, and significance thresholds. Captions will distinguish individual evidence trajectories, averages, and rejection probabilities and define the displayed uncertainty. We will also replace the non-rejection label in Figure 1 with “inconclusive within the available budget.”
5. Elevated FPRs for DICE and PETAL
We will investigate both DICE's 0.087 and PETAL's 0.644, including the shared testing backend and method-specific feature generation, calibration, and preprocessing. The checks will compare the original procedure with unprocessed features, independently calibrated preprocessing, and strictly past-based shared preprocessing. We will repeat null trials, report rejection counts and uncertainty intervals, and rerun affected comparisons. Weak features alone do not explain inflated false rejection under a valid null procedure.
6. Anytime validity of preprocessing and trimming
The trimming description leaves unclear whether current observations influence the thresholds or replacement values. We will make these dependencies explicit and ensure that the common transformation, witness, and stake are fixed before observing each new pair.
Under conditional pair exchangeability, swapping the pair reverses the bounded discrepancy score while leaving those choices unchanged, giving zero conditional mean. With admissible stakes, the resulting nonnegative wealth process supports the anytime error bound. We will verify these conditions for the complete implementation, including swap symmetry and independence from future data, and rerun affected experiments.
7. Purpose of Table 2
Table 2 qualitatively illustrates the continuation interface using the same prefix and outputs from different APIs. It does not establish training membership; a plausible continuation can arise through generalization. We will clarify its caption and role in Appendix A.6. The planned API analysis will report systematic continuation-compliance measurements.