acceptodds
Under review as a conference paper at ICLR 2027

From Membership Scores to Unlearning Claims in Offline Reinforcement Learning

Abstract

Machine unlearning aims to remove the influence of selected training data by matching a model retrained without them. Membership scores offer a convenient evaluation tool, but interpreting their changes requires knowing which properties of retraining they measure. We study this connection through controlled experiments in offline reinforcement learning, where simulator access and training records allow us to separate policy compatibility from training exposure. Naturally generated non-members receive member-like scores under a fixed audit, while interventions on reference selection and target adaptation reveal how additional information changes the conclusions of the same score family. We then examine whether score-based comparisons can recognize retain-only retraining itself. Two candidate audit designs frequently reject retraining controls despite consistently rejecting full-data comparators, showing why comparator separation alone does not validate an equivalence test. We explain these findings by distinguishing model-distribution distance from distances between individual models and by characterizing information lost in finite action-error scores. A three-action decision rule connects valid uncertainty bounds to support, rejection, or abstention, and a practical protocol specifies the claim, access assumptions, independent samples, and controls needed to apply it. The resulting case study shows how membership evidence can remain useful for diagnosis while supporting retraining-equivalence claims only through an explicitly validated statistical connection.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Unlearning, Offline Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3UBfOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This work studies on what membership-style evidence can and cannot establish in machine unlearning. It makes a controlled study of what a standard action-error membership score measures, a test of whether score-based rules can recognize retain-only retraining, and a statistical framing of a hard-pair total-variation bound on joint coverage. The central negative result is well-designed.

Critical strengths

The paper separates three statistical objects that unlearning audits routinely conflate: a model distribution, a fixed checkpoint, and the trajectory scores a checkpoint produces. It shows that the pairwise tagged-KS distance is only an upper-bounded surrogate for distributional distance in Eq. (5). The empirical consequence is decisive for a currently popular evaluation practice. The combination of “the rule rejects the bad thing” and “the rule also rejects the good thing” demonstrates the failure mode that per-task unlearning papers do not check.

Critical weaknesses
  1. The gap to prior work is not established where it matters most, which makes it hard to judge whether the contribution is a discovery or a confirmation. The paper should states the difference among Zhang et al. (2025), Yang & Yeung (2026), and this paper.
  2. The empirical base is narrow. All results are on two MuJoCo locomotion tasks. As the paper mentioned, “these are resampling slots, not additional independent trained models” and “repeated resampling cannot replace additional independent models”. The intervals reflect trajectory-level resampling rather than the model-level variability that the claim is about. Similarly, the strongest new evidence where 10/10 secret-reference transfers is established with only 4 secret checkpoints per environment.
Questions for authors
  1. Could you report a model-level sensitivity analysis, e.g. leave-one-model-out across the three sources per role, or the spread of the positive-pass probability across the individual source combinations, so that the 0.134/0.289 estimates are not presented solely with resampling-level uncertainty?
  2. This work argues that “rejecting full-data comparators coexists with frequent rejection of retraining controls”. Is there a decision rule you would recommend instead that reports diagnostics without inviting the equivalence interpretation?
Reviewer_wWKTOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether membership scores can reliably evaluate machine unlearning in offline reinforcement learning. Through controlled experiments, it demonstrates that non-member trajectories can produce member-like scores and that two candidate auditing methods frequently reject valid retraining controls. The authors further propose a statistical framework distinguishing membership evidence from retraining equivalence.

Critical strengths
  1. The controlled experiments effectively demonstrate how non-member trajectories and adaptive actions can produce misleading membership evidence.

  2. The distinction between pairwise model distances and distributional equivalence is clearly explained, providing useful guidance for unlearning evaluation.

Critical weaknesses
  1. Limited novelty. The central conclusions substantially overlap with existing research on membership inference limitations and retraining-based unlearning evaluation. The theoretical results mainly rely on standard statistical principles.

  2. Insufficient experimental validation. The main retraining-equivalence experiments consider only two heuristic auditing designs and three independent model sources per role. The paper does not establish whether the observed failures generalize to stronger auditing methods or demonstrate a practical calibrated audit for deep RL.

Questions for authors
  1. What are the key contributions beyond existing retraining-grounded auditing frameworks, particularly Yang and Yeung (2026)?

  2. Would the reported failures persist with more independent retraining runs and statistically validated auditing methods?

  3. Can the framework be evaluated more comprehensively on practical offline RL unlearning algorithms?

Reviewer_KNSiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates what membership-inference scores can or cannot establish about machine unlearning, using offline RL (D4RL Walker2d / HalfCheetah, TD3+BC policies, action-error scores) as a controlled testbed. It shows that 1) natural non-member trajectories frequently receive member-like scores at a fixed threshold; 2) provenance-aware reference selection removes some non-adaptive confounds, but action-adaptive targets still transfer to secret references, and finite score vectors are non-injective (Proposition 1); 3) two candidate score-based audit rules reliably reject full-data comparators but also reject retain-only retraining controls too often. The paper argues that averaged pairwise distances are not model-distribution distances, provides a three-action decision framework, and proposes a protocol card for unlearning audits.

Critical strengths

Clear conceptual distinctions with carefully controlled experiments. The paper cleanly separates historical exposure from retraining equivalence, and pairwise model distances from distances between model distributions. Using retain-only retrains as positive controls, rather than only checking that full-data models are rejected, is a useful methodological point for the unlearning-evaluation community.

Critical weaknesses
  1. Limited novelty relative to existing work. As acknowledged in the paper, the key insight, that membership scores do not by themselves prove training participation or unlearning, and that evaluation should be grounded in retraining controls with abstention, has already been studied in Zhang et al. (2025), Tu et al. (2025), and Yang & Yeung (2026). The theoretical results (TV bound, confidence-set decision rule, Hoeffding interval) are standard statistical facts. The main contribution of this paper is the empirical case study. However, it is not strong enough on its own.

  2. Insufficient empirical justification that does not support the general claim. The claims are framed generally (membership scores vs. unlearning claims), but the evidence uses a single, relatively weak, RL-specific score, two locomotion environments, one policy class, and only three independently trained models per role. The two audit rules shown to fail are constructed by the authors rather than methods widely used in practice. There are no experiments on the standard unlearning benchmarks (e.g., CIFAR-10/100 image classification, or LLM benchmarks such as TOFU/MUSE), and no comparison with strong, calibrated MIAs such as LiRA / U-LiRA. Closely related work such as [R1] , is not cited. As a result, it is unclear whether the findings reflect a general limitation of membership-based unlearning evaluation or are specific to this weak score and setting.

[R1] Hayes et al. (2024), "Inexact Unlearning Needs More Careful Evaluations to Avoid a False Sense of Privacy"

Questions for authors
  1. Do the main findings hold for a stronger, calibrated MIA such as LiRA/U-LiRA, and on a standard benchmark such as CIFAR-10 with ResNet-18?

  2. Image classification also permits retraining and has complete training records. Why is offline RL necessary for the argument, and which conclusions are specific to RL, e.g., policy generalization making unseen trajectories member-like?

  3. With three models per role, the Hoeffding-based interval is acknowledged to be uninformative. How many independent retrains would the proposed protocol require in practice for a decisive conclusion, and is this cheaper than simply retraining?

  4. How does the work differ concretely, in findings rather than framing, from Yang & Yeung (2026) and Hayes et al. (2024)?

Comments on the paper

2 comments

Thank you for reconsidering the score, and for the question that led to the correction.

We agree that the original development study should have caught this, and we would like to be precise about what changed. On the same simulated draws, correct accounting moves the tagged-KS outcomes for retraining controls from 0.237/0.381/0.382 to 0.143/0.842/0.015 (authorize/abstain/reject; Table 2). Rejections become abstentions, not authorizations. With three retrains, the rule still does not recognize retraining, so that part of the submitted conclusion stands. What changed is the mechanism and the failure mode: the rule does not wrongly reject retraining because of the information in the score, but cannot decide because three retrains are too few. We withdrew the earlier attribution, and the revised abstract and introduction state the correction.

The original validation missed the error for the reason the paper warns about. It checked that full-data models were rejected, and they were rejected under both accountings. Only the retraining controls, examined at the model level as you asked, exposed the defect.

The revised evidence no longer depends on that development study. We tested the mechanism on 67 newly trained independent models. Imposing the slot construction reproduces the failure (rejection 0.498), and source-level accounting removes it (0.044) (Table G.3). All later results use independently trained models as units, with whole-model permutation tests and model-bootstrap intervals. The discussion-period analyses fixed their gates and predictions before scoring.

Thank you for engaging with the response and for acknowledging the added experiments. We agree that retraining controls, calibration and abstention as principles have precedents. We restate three findings that are not reported in Zhang et al., Tu et al., Yang & Yeung or Hayes et al., and that would each change how an audit is run or read.

  1. The auditor used to evaluate a published offline-RL unlearning method tracks lineage far more than exposure. Under TrajAuditor, full-data models from other seeds, which trained on every forget trajectory, are judged nearly as forgotten as retrains (persistence 0.43 vs 0.38). Our port of TrajDeleter is judged more forgotten than genuine retraining (0.04), while a model-level test separates it from retraining (Appendix G.8). This is an empirical result about an existing evaluation, not a reframing.
  2. A rule that is already retrain-relative and rejects every full-data model can still reject genuine retraining, purely through sample accounting. Yang & Yeung show that oracle retrains can fail absolute rules. Here the failure has a different cause and a different fix: counting reused retrains as independent models (rejection 0.382 → 0.015 on the same draws; 0.498 vs 0.044 on 67 new independent models; Tables 2 and G.3).
  3. Policy compatibility reverses raw membership evidence, and what survives calibration depends on training length. In all 107 Lift models, unseen policy-generated trajectories rank above the training demonstrations. After calibration, exposure is identified with AUC near 0.95 at 200k steps but only 0.60–0.79 at 5k–20k steps (§3.1, Table G.6).

We respect your assessment and thank you again for the questions that sharpened the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.