acceptodds
Under review as a conference paper at ICLR 2027

The Scar of RL: Membership Inference Attack for Reasoning Models

Abstract

Reinforcement learning (RL) is a critical post-training stage of frontier reasoning models. However, the prompts used during RL training are rarely disclosed, making it difficult to determine whether benchmark questions used to evaluate post-trained models were included during RL training, and potentially biasing downstream evaluations. Existing membership inference attacks, developed primarily for pretraining and supervised fine-tuning, typically exploit the reduced loss of an observed training sequence. However, this principle does not transfer directly to RL where there are no observed sequences. Instead, RL optimizes for rewards given to sampled outputs and reshapes the distribution over model-generated solution trajectories. We show that this change leaves a multi-dimensional signature in the model’s own outputs as prompts seen during RL tend to induce higher confidence in sampled continuations, more similar solution trajectories and final answers, more saturated verifier outcomes, and less variation across independent rollouts. However, the magnitude of these effects varies substantially across model–dataset pairs, making any single statistic unreliable. In response, we introduce SCAR, which defines 15 features capturing complementary signatures of RL exposure, and combines them through label-free multi-axis aggregation, avoiding reliance on any single signature. Across eleven unsupervised aggregation rules, combining the four axes (Self-sharpening, Collapse, Answer saturation, and Rollout-sharpening) consistently improves macro-average performance over individual signatures. Our strongest aggregation reaches mean ROC-AUC, compared with for the strongest single-statistic baseline.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Language models, Reinforcement Learning, Data contamination, Membership Inference Attack

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_LLpaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Texts that appear in the training corpora of unsupervised or supervised learning algorithms tend to be modeled with higher likelihood, and can therefore be distinguished from novel texts at rates better than chance. This manuscript studies the problem of detecting prompts that have been used in RL training procedures based on the likelihoods and distribution of the rollouts they cause a model to generate. Eleven diagnostic features in four categories are proposed: Sharpened likelihood distribution within a rollout, Trajectory collapse, Answer saturation, and sharpened likelihood distribution across Rollouts.

Critical strengths
  • The four feature categories are each well-motivated by existing literature and theory. The eleven features that compose them are certainly not comprehensive, but are well-motivated representative examples of each category.
  • Experimental results demonstrate the claim that no single feature category or single baseline provides detection AUC better than a combination of the four feature categories.
  • The label-free multi-axis weighted aggregation algorithm consists of a well-motivated initialization and a well-motivated refinement, though both are marred by heuristic hyperparameters.
  • Robustness analysis shows that the four-axis feature combination outperforms any single feature even if features are uniformly combined, suggesting that the algorithm is robust to the hyperparameters of the label-free weight estimation.
  • In each of the four controlled experiments, half of a benchmark's prompts are included in training data, and half are not, so there is no systematic difference between in-training and non-training prompts other than whether or not they were used in RL training. Equivalence of distribution between in-training and non-training prompts is further verified by the near-chance performance of SCAR using the pre-RL version of each model.
Critical weaknesses
  • The refinement stage of the label-free refinement algorithm assumes that the top-scoring and bottom-scoring 30% from the first stage are mostly true positives and false positives, respectively. This hyperparameter might be reasonable if exactly 50% of all tested prompts were in the training data, but is unlikely to be reasonable in any real-world deployment.
  • The distribution of ground truth labels would also affect the thresholds used for detection. Even with uniform weights across SCAR, how robustly can thresholds be estimated if ground truth positive rate is much different from 50%?
  • Although SCAR performs 0.09 better than the best baseline, its AUC is still only 0.73. Isn't that still too low to be useful for the in-training vs. out-of-training diagnosis of a real-world benchmark?
Questions for authors
  • Some of the measures are defined too succinctly to be intelligible, e.g., what is mink20(Li)?
  • Can you run experiments with a ground truth distribution substantially different than 50%, to judge model robustness?
2 comments

We thank the reviewer for their review. Below, we address the concerns and questions raised.

W1/W2/Q2: Refinement and contamination rates

The top- and bottom-scoring 30% of the audit pool are used as pseudo-seen and pseudo-unseen groups for refinement. This procedure does not assume a 50% membership rate, although highly imbalanced pools may affect its performance.

To directly address these concerns, Appendix F.3 (Figure 9) evaluates SCAR-Cal and SCAR-Uniform across contamination rates ranging from 15% to 85% on OLMo/GSM8K and DeepSeek/AIME25. Both methods maintain informative membership rankings across the tested rates without requiring knowledge of the true membership proportion.

For extreme cases where the audit pool contains only members or only non-members, a practical approach is to augment the pool with reference questions known to be members and non-members, when available, and rerun the audit. This introduces membership variation into the pool and avoids contamination rates near 0% or 100%, although it requires access to suitable reference questions.

W3: Practical usefulness

We would like to clarify that SCAR-Cal achieves a mean ROC-AUC of 0.764, not 0.73, compared with 0.673 for the strongest baseline, Min-kNN. SCAR-Uniform achieves 0.737.

Although this does not guarantee correct membership decisions for every question, it provides more informative rankings for identifying potentially contaminated benchmark questions. Furthermore, all eleven label-free aggregation rules outperform the strongest baseline in macro-average ROC-AUC (Figure 3).

Q1: Signature definitions

We thank the reviewer for highlighting the need for clearer notation. The signature mink⁡20(Li)\operatorname{mink}_{20}(L_i) denotes the average log-probability of the least probable 20% of tokens in rollout ii. We then average this quantity across the NN sampled rollouts to obtain the signature for each question. Complete notation and definitions are provided in Appendix A.1.

We hope our responses have addressed your concerns and clarified the points raised in your review.

Dear Reviewer,

As the discussion period ends in two days, we kindly wanted to follow up on our rebuttal. We would greatly appreciate it if you could review our responses and share any remaining concerns before the deadline.

Thank you for your time and consideration.

Reviewer_zLwyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This article introduces SCAR, which uses four types of features — self-confidence, trajectory collapse, answer saturation, and cross-rollout stability — to detect whether a question was ever used in RL training for a reasoning model, all without needing membership labels.

Critical strengths
  1. The paper investigates an important issue that hasn’t received much attention before: traditional membership inference relies on the likelihood of known training sequences, while in the RL stage, only the prompt is fixed and the response is generated online by the policy. So existing attacks can’t be directly transferred. I like that the paper describes RL exposure as multidimensional variations over the rollout distribution, instead of forcing a single statistic corresponding to training loss.

  2. SCAR’s four-axis design is pretty clear, covering token confidence, reasoning trajectory concentration, verifier outcome saturation, and cross-sample stability. The authors further show that the most effective axis differs across model-dataset combinations, so aggregating multiple signals makes sense. The 11 unsupervised aggregation methods generally outperform the strongest single statistic, which also suggests that the gain doesn’t entirely depend on some complex aggregation rule.

Critical weaknesses
  1. The construction of members and non-members on public models lacks sufficient detail, and it’s also impossible to strictly attribute the detection signals to RL. The appendix only mentions using 300 released RL prompts and 300 unseen prompts for each public model, but it doesn’t specify the source of the unseen prompts, their difficulty matching, topic distribution, or deduplication method. Since these models don’t have matching pre-RL checkpoints, SCAR might partly rely on differences caused by data sources, question difficulty, pretraining exposure, or other post-training stages. The four controlled experiments help with this issue, but three of the AIME experiments only have about 13–15 member samples, which isn’t enough to independently support broad conclusions on public models.

  2. SCAR is essentially a transductive pool-level ranking rather than a deployable single-sample membership test. Feature normalization, Gaussian mixture separation, axis weighting, and top/bottom 30% pseudo labels all rely on having enough member–non-member variation in the candidate pool. When the candidate pool is almost entirely members or non-members, the algorithm is still forced to produce relative high or low scores; the paper also doesn’t provide a decision threshold that can be chosen without membership labels. So, while ROC-AUC can indicate ranking ability, it can’t directly achieve the practical goal mentioned in the introduction of ‘determining whether a benchmark question was used in training.’

Questions for authors

See the weaknesses above.

5 comments

We thank the reviewer for their review. Below, we address the concerns raised.

W1: Membership provenance, distributional differences, and limited AIME sample sizes

Public checkpoints and RL-stage isolation

Since the public models were trained on the full released RL datasets, constructing member and non-member splits from the same dataset is not possible. We therefore follow the evaluation protocol of Min-kNN, using 300 RL-training members and 300 non-members per public model. The non-member pool consists of 100 BeyondAIME, 100 Omni-Math, 40 MATH-500, 30 AIME 2024, and 30 AIME 2025 questions. We also checked for duplicate questions between the member and non-member pools and found none.

We acknowledge that differences in problem source or difficulty may contribute to the separation observed for public checkpoints. To investigate whether the membership signal is attributable to RL training, we additionally conduct controlled RL experiments using matched pre-RL policies. As shown in Table 3, SCAR-Cal performs near chance before RL training, whereas membership separation becomes substantially stronger afterward. These results provide evidence that the observed separation is not explained solely by pre-existing distributional differences between seen and unseen questions.

AIME sample sizes and robustness

We acknowledge that the controlled AIME experiments have relatively small evaluation pools. To account for the uncertainty associated with these sample sizes, we report ROC-AUC together with the standard deviation obtained from 4,000 bootstrap resamples of the evaluation questions. This makes the uncertainty of the reported results explicit rather than relying solely on point estimates.

To further investigate robustness to rollout sampling, we conducted an additional experiment with 32 independent repetitions, each using eight newly sampled rollouts per question. In each repetition, we recomputed the 15 signatures and SCAR-Cal membership scores and evaluated ROC-AUC. The resulting performance is following:

BenchmarkSCAR-CalSCAR-Cal min–maxSCAR-UniformMin-kNN
JustRL0.7098 ± 0.00330.7043–0.71660.7042 ± 0.00580.6134 ± 0.0094
DAPO0.8052 ± 0.00470.7945–0.81530.7900 ± 0.00530.7336 ± 0.0057
SimpleRL0.7360 ± 0.00560.7263–0.74760.7381 ± 0.00590.7249 ± 0.0069
ORZ0.8202 ± 0.00450.8093–0.83000.8086 ± 0.00500.7985 ± 0.0065
OLMo / GSM8K0.6890 ± 0.00760.6744–0.70450.6819 ± 0.00690.6385 ± 0.0091
Qwen / AIME250.6420 ± 0.03950.5590–0.70260.6446 ± 0.03220.5713 ± 0.0465
Qwen / AIME260.7129 ± 0.04090.6267–0.80440.7078 ± 0.03560.7101 ± 0.0370
DeepSeek / AIME250.6829 ± 0.05690.6000–0.78970.7021 ± 0.06190.5325 ± 0.0634
Macro average0.7247 ± 0.00900.7106–0.74690.7221 ± 0.00770.6654 ± 0.0114

These additional results demonstrate that SCAR-Cal retains membership discrimination across different rollout samples, while also revealing greater variability in the smaller AIME evaluation pools. The bootstrap uncertainty and repeated rollout sampling do not replace evaluation on larger independent datasets, but they provide a more complete assessment of the reliability of our findings.

W2: Transductive ranking, audit-pool composition, and membership decisions

Transductive setting and pool composition

We agree that SCAR is a method that operates over a pool of candidate questions. However, we explicitly evaluate its sensitivity to pool composition in Appendix F.3, varying contamination rates from 15% to 85% and considering different pool sizes. The results show that SCAR-Cal maintains informative ranking performance across the tested settings without requiring knowledge of the contamination rate.

For extreme cases where the pool contains only members or non-members, a practical approach is to augment it with reference questions from both groups, where such questions are available, and rerun the audit. This can introduce the variation required for aggregation.

Membership scores versus binary decisions

We intentionally formulate SCAR as a membership-ranking method rather than a binary classifier, consistent with prior membership inference attacks such as Min-K% (Shi et al., 2024), Min-K%++ (Zhang et al., 2025), and the RL-specific Min-kNN (Zhang et al., 2026), which evaluate membership scores using ROC-AUC.

Converting these scores into binary predictions requires selecting a threshold, which can be calibrated using labeled examples from other benchmarks or chosen to meet the auditor's desired FPR/TPR operating point, provided suitable calibration data are available. We therefore focus on evaluating membership-ranking performance without assuming a particular threshold or calibration setting.

We hope these clarifications address your concerns, and we would greatly appreciate it if you could reconsider your score in light of our responses and additional evidence.

Dear Reviewer,

As the discussion period ends in two days, we kindly wanted to follow up on our rebuttal. We would greatly appreciate it if you could review our responses and share any remaining concerns before the deadline.

Thank you for your time and consideration.

Reviewer_S4vMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SCAR studies prompt membership in an audited RL training stage using newly sampled solutions rather than an observed training response. It combines 15 statistics into four axes: within-rollout confidence, trajectory collapse, answer saturation, and across-rollout consistency. SCAR-Cal weights standardized axes using agreement and mixture separation, then refines a score using pseudo-labeled tails. Across four public checkpoints and four controlled GRPO settings, the reported mean ROC-AUC is 0.7636 versus 0.6735 for Min-kNN; uniform axis averaging already reaches 0.7370. Matched starting-policy controls and GSM8K checkpoint trajectories support an exposure-related signal. The contribution is chiefly the multi-signal representation and its empirical complementarity; structural convergence and self-critique entropy signals already have direct precedents.

Critical strengths
  1. The evidence for combining complementary signals is useful and substantially stronger than a single winning configuration. All eleven fusion rules improve macro-AUC over the strongest individual axis, uniform averaging retains much of the gain, and removing any axis reduces the full method's mean AUC (Tables 3-4, 7, 10). The result therefore supports the proposed representation even without accepting every detail of SCAR-Cal. Gains also persist on the four larger public pools: recomputing Table 10 gives 0.7732 versus 0.7322 for Min-kNN.

  2. The controlled starting-policy comparison is an important safeguard against attributing any confidence difference to the audited RL stage. Aggregate AUC moves from 0.553/0.523/0.533/0.492 before the additional training to 0.691/0.687/0.787/0.851 afterward (Tables 1, 15). Appendix F.6 appropriately acknowledges pre-existing axis-level separation. The full feature trajectories and the openly reported failure under strong rewriting further improve the informativeness of the empirical study.

Critical weaknesses
  1. Membership provenance and stage isolation remain insufficiently auditable. Appendix D states that public-checkpoint pools contain 300 members and 300 non-members, but does not identify the non-member sources, matching procedure, overlap criterion, or exact released-data revisions. Since several axes also reflect difficulty, source differences could produce part of the public-checkpoint separation; the manuscript explicitly lacks matched starting-policy controls for these four models. The controlled AIME25 construction also needs reconciliation: Appendix D excludes four of 30 problems, whereas Figure 12 labels 13 seen and 15 unseen problems, totaling 28. Appendix C supplies many hyperparameters but not the AIME training duration or exposure counts. These are consequential gaps in interpreting a stage-specific membership result, not evidence of fabrication.

  2. The strongest robustness claims are less secure than the headline average suggests. AIME pools have only 13-15 examples per class, and Table 1 gives AUC SDs up to 0.106 for SCAR-Cal. Question bootstraps do not establish stability across training splits or rollout draws, and it is unspecified whether each transductive bootstrap refits normalization, mixtures, and pseudo-tail refinement. Section 4 selects SCAR-Cal for its best reported macro-average, while Table 8's LOPO result falls from 0.8513 to 0.6359 on DS-Math/AIME25; thus label-free application should be distinguished from independently validated method selection. Appendix F.2 addresses parameter sensitivity but does not eliminate this concern. Moreover, Table 2's 5%-FPR advantage is concentrated in the tiny AIME pools; SCAR-Cal trails the best baseline on all four public pools. Fifteen non-members give empirical FPR increments of 6.7%, so those AIME results do not establish reliable 5%-FPR calibration.

Questions for authors
  1. How does the audit distinguish RL membership from differences in problem source or difficulty? For the public checkpoints, which matching controls or comparisons support the claim that SCAR is detecting training exposure rather than generally easier questions? Please also reconcile the AIME25 sample counts and explain whether the main conclusions change under the consistent sample definition.

  2. How sensitive are the gains to choosing the default fusion rule on the evaluated audit settings? In the reported bootstrap, is the full transductive estimator refitted, including normalisation, mixture fitting, and pseudo-tail refinement? Please explain how the DS-Math leave-one-pipeline-out drop and the small AIME pools affect the claims about robustness and low-false-positive detection.

  3. What access and sampling costs are needed to obtain the reported advantage? Table 4 already shows that the correctness axis is not the sole source of improvement; how should full SCAR be compared with sampling-only or probability-based baselines under comparable generation budgets? For the rewriting experiment, please clarify which question version is used for RL and which is audited, since this determines what kind of robustness is being tested.

6 comments

We thank the reviewer for their thoughtful review. Below, we address the concerns and questions raised.

W1/Q1: Membership provenance, distributional differences, and training details

Public checkpoints and distributional differences

Since the public models were trained on the full released RL datasets, constructing member and non-member splits from the same dataset is not possible. We therefore follow the evaluation protocol of Min-kNN, using 300 RL-training members and 300 non-members per public model. The non-member pool consists of 100 BeyondAIME, 100 Omni-Math, 40 MATH-500, 30 AIME 2024, and 30 AIME 2025 questions. We also checked for duplicates between the member and non-member pools and found none.

We acknowledge that differences in problem source or difficulty may contribute to the separation observed for public checkpoints. This is why we also conduct controlled RL experiments and examine the matched pre-RL policies. As shown in Table 3, the aggregate membership signal is near chance before RL and becomes substantially stronger after training, providing evidence that the observed separation is not explained solely by pre-existing distributional differences. Furthermore, Figure 13 shows that the discriminative power of individual signatures varies considerably across model–dataset pairs, supporting our motivation for combining complementary signatures rather than relying on any single one.

AIME 2025 sample counts

We thank the reviewer for pointing out the ambiguity in the AIME 2025 evaluation pools. Each of the two main AIME 2025 pools contains 13 members and 15 non-members, totaling 28 questions. Two member questions were excluded due to overlap with NuminaMath. We have clarified the resulting sample counts in the revised manuscript.

Training duration

Qwen/AIME25 and Qwen/AIME26 were trained for 3 epochs, while DeepSeek/AIME25 was trained for 4 epochs, reflecting differences in model size.

W2/Q2: Hyperparameter transfer and robustness

Hyperparameter transfer

Leave-one-pipeline-out (LOPO) evaluation examines whether aggregation hyperparameters selected on seven pipelines generalize to the held-out pipeline. Our main experiments use the same fixed configuration, (ρ,λ)=(0.30,0.10)(\rho,\lambda)=(0.30,0.10), across all eight pipelines.

For DeepSeek/AIME25, ROC-AUC decreases from 0.8513 with the fixed configuration to 0.6359 under LOPO selection. This indicates that hyperparameters selected on other pipelines do not necessarily transfer well to a new setting, potentially reflecting overfitting to the characteristics of the source pipelines. We will clarify this interpretation and the diagnostic purpose of LOPO in the revised manuscript.

For practical auditing, we believe that examining sensitivity across a range of hyperparameters provides a more informative assessment of robustness. The following table reports the mean ROC-AUC across all eight pipelines for different values of (ρ,λ)(\rho,\lambda). The 15 signatures remain fixed throughout, and only positive covariance-shrinkage values are considered.

ρ\λ\rho\backslash\lambda0.050.100.250.501.00
0.100.71960.72400.72510.72860.7395
0.200.73750.73740.74220.74120.7368
0.300.76440.76360.76200.75910.7519
0.400.74910.74900.74880.74920.7456

These results support assessing hyperparameter sensitivity over a range of configurations rather than relying exclusively on LOPO selection.

Bootstrap evaluation and robustness to rollout sampling

Our original 4,000 bootstrap resamples estimate question-sampling uncertainty conditional on fixed membership scores. Since SCAR-Cal estimates normalization and aggregation from the audit pool, these intervals do not capture uncertainty from refitting the entire aggregation procedure. To additionally examine sensitivity to rollout sampling, we repeated the evaluation 32 times, independently generating eight rollouts per question in each repetition and recomputing the 15 signatures and membership scores. We report the mean, standard deviation, minimum, and maximum ROC-AUC across these repetitions.

BenchmarkSCAR-CalSCAR-Cal min–maxSCAR-UniformMin-kNN
JustRL0.7098 ± 0.00330.7043–0.71660.7042 ± 0.00580.6134 ± 0.0094
DAPO0.8052 ± 0.00470.7945–0.81530.7900 ± 0.00530.7336 ± 0.0057
SimpleRL0.7360 ± 0.00560.7263–0.74760.7381 ± 0.00590.7249 ± 0.0069
ORZ0.8202 ± 0.00450.8093–0.83000.8086 ± 0.00500.7985 ± 0.0065
OLMo / GSM8K0.6890 ± 0.00760.6744–0.70450.6819 ± 0.00690.6385 ± 0.0091
Qwen / AIME250.6420 ± 0.03950.5590–0.70260.6446 ± 0.03220.5713 ± 0.0465
Qwen / AIME260.7129 ± 0.04090.6267–0.80440.7078 ± 0.03560.7101 ± 0.0370
DeepSeek / AIME250.6829 ± 0.05690.6000–0.78970.7021 ± 0.06190.5325 ± 0.0634
Macro average0.7247 ± 0.00900.7106–0.74690.7221 ± 0.00770.6654 ± 0.0114

The public checkpoints exhibit low sensitivity to rollout sampling, with ROC-AUC standard deviations between 0.0033 and 0.0056. The controlled AIME experiments show greater variability, consistent with their smaller evaluation pools. At the eight-rollout budget, SCAR-Cal achieves the highest macro-average ROC-AUC among the three evaluated methods (0.7247 versus 0.7221 for SCAR-Uniform and 0.6654 for Min-kNN).

In addition, Table 10 in Appendix F.5 reports the ROC-AUC of the different aggregation rules across individual benchmarks, providing further evidence of their performance across model–dataset settings.

Performance at low false-positive rates

We compute TPR at a specified false-positive rate by selecting the maximum empirical TPR subject to an FPR no greater than the target. Tied scores are retained, and no interpolation is applied. For the AIME evaluation pools containing 15 non-members, the empirical FPR changes in increments of 1/151/15, or approximately 6.7 percentage points. Consequently, at a target FPR of 5%, no observed false positives are permitted. We emphasize that this finite-sample operating point does not establish population-level false-positive-rate calibration.

Q3: Access requirements, rollout budget, and prompt rewrites

Access requirements and computational budget

The full SCAR-Cal framework requires 16 sampled responses per prompt, access to sampled-token log-probabilities, and a task verifier. Using all four axes, SCAR-Cal achieves a macro-average ROC-AUC of 0.764. To evaluate the contribution of each access requirement, we also consider restricted settings. Removing access to the verifier reduces ROC-AUC to 0.725, while removing token log-probabilities reduces it to 0.709. Using only the text-based Collapse (C) axis yields an ROC-AUC of 0.680.

For comparison, using the same budget of 16 sampled rollouts per prompt and averaging the PPL and Min-K20 scores across these rollouts, PPL achieves mean ROC-AUC of 0.682, Min-K20 achieves 0.684, and Min-kNN achieves 0.673.

Prompt-rewriting experiments

In our prompt-rewriting experiments, RL training is performed exclusively on the original questions. After training, we apply the rewriting transformations to both the seen and unseen question sets and evaluate membership inference on the resulting rewritten prompts. Importantly, the policy is not retrained on any of the rewritten questions. Thus, these experiments evaluate whether membership signatures remain detectable under prompt reformulations, rather than measuring the effect of training on the rewritten prompts themselves.

We hope our responses have addressed your concerns, and we would greatly appreciate it if you could reconsider your score in light of these clarifications and additional results.

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

We thank you for your time and feedback. As the discussion period concludes, we would like to briefly summarize our responses and additional evidence.

Our evaluation on public RL-trained checkpoints aims to demonstrate the applicability of SCAR across diverse, realistic RL settings. Since these models were trained on the full released datasets, constructing exactly distribution-matched member and non-member pools is not feasible. This is a general challenge in evaluating public checkpoints, rather than a limitation specific to SCAR.

For controlled experiments, it is also essential to select non-member questions that we can confidently establish were not used during earlier training stages. This limits the choice of suitable benchmarks and motivates our use of recent AIME problems rather than larger datasets with uncertain prior exposure. To address concerns about the small AIME evaluation pools, we additionally conducted a larger controlled experiment using OLMo on GSM8K, with 200 members and 200 non-members. Our matched pre-RL controls further show that the aggregate membership signal emerges primarily after RL training (which shows that SCAR is detecting RL exposure and not distribution shift).

Beyond the extensive ablations and robustness analyses already included in the paper, we provided additional experiments on rollout-sampling variability, hyperparameter sensitivity, and different contamination rates. We also conducted new AIME experiments varying the source of non-member questions across AIME 2025 and 2026, further supporting the benefit of multi-axis aggregation. Regarding hyperparameter tuning, we evaluated a broader grid of refinement and covariance-shrinkage parameters and show that the method is robust to different hyperparameters.

We also acknowledged the limitations of low-FPR evaluation on the small AIME pools. With only 15 non-members, the empirical FPR changes in increments of approximately 6.7%, making reliable conclusions at 5% FPR difficult. We therefore clarified this limitation and will restrict low-FPR claims to evaluation pools sufficiently large to support them.

Importantly, our central contribution is not the superiority of a particular aggregation rule, but demonstrating that combining complementary RL membership signatures improves performance over individual signatures and existing baselines across the evaluated settings.

We hope these clarifications and additional results will be considered in the final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.