acceptodds
Under review as a conference paper at ICLR 2027

Suppression Is Not Deletion: Evaluating Offline-RL Unlearning Against Retraining

Abstract

Deleting trajectories from an offline reinforcement learning dataset should leave the policy that training without them would have produced. Rather than retrain, several published offline-RL unlearning methods keep the deleted trajectories in the update and push the learner away from them, which we call suppression. This paper contributes a behavioral audit of unlearning against retraining, a per-state analysis of what suppression objectives optimize, and an evaluation of published methods on the benchmark released with TrajDeleter. Since retraining is random, a fresh seed giving a different policy, the audit compares a candidate with a family of independent retrains, and first audits the unchanged policy, the original with no unlearning applied, because if it passes, a candidate's pass proves nothing about deletion. At the benchmark's own deletion sizes the unchanged policy behaves like the retrained policies wherever the audit is validated on held-out retrains, so a pass there cannot show that a method deleted anything. On larger HalfCheetah deletions that do change behavior, no suppression method we evaluate reliably reaches the retrain family, whereas retrained policies stay inside it under further retained-data training. The analysis explains why: deletion gives the removed data zero weight, negative weighting of them overshoots the retraining target, and negating their rewards changes what the value function, the critic, learns; tests declared in advance find each effect in the predicted direction on HalfCheetah. The benchmark's own forgetting score follows the critic's values rather than the policy's behavior: shifting the critic by a constant reports complete forgetting while every action stays unchanged. Unlearning in offline RL should be judged against the behavior of retraining.

Primary area:
reinforcement learning
Keywords:
machine unlearning, offline reinforcement learning, data deletion, retraining reference, actor–critic, behavioral audit, occupancy measure, forgetting scores, TrajDeleter, evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_cVUZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines whether existing offline reinforcement learning unlearning procedures reproduce the behavior of training from scratch on the retained data. It proposes an audit based on discounted state–action occupancies, using independent retraining runs to define a reference family and an unchanged-policy control to check whether a deletion is detectable before interpreting an unlearning method's acceptance. On the TrajDeleter benchmark's 1% and 5% episode deletions, the unchanged policies usually pass wherever the retraining reference passes the authors' calibration check, making successful unlearning difficult to establish with this audit. For the informative 30% HalfCheetah deletions, the evaluated suppression procedures generally fail to match retraining, including additional runs of the released TrajDeleter procedure and a reward-hinge port. A per-state analysis attributes possible failures to negative actor weighting, amplification from balanced minibatches, and reward negation in the critic. The paper also demonstrates that constant shifts of a frozen policy's critic can drive TrajAuditor's forgetting score to zero without changing any action. The contribution is a critique of current evaluation practices and an accompanying behavioral audit.

Critical strengths
  • The unchanged-policy control exposes an important weakness in unlearning evaluation. In the calibrated benchmark settings, only one of 27 unchanged-policy evaluations is rejected. Checking this baseline first prevents acceptance of an edited policy from being mistaken for evidence of deletion when doing nothing would also pass.

  • The critic-shift experiment is particularly convincing. Keeping the actor fixed while shifting the critic makes all 39 tested evaluations report complete forgetting at the stated shift magnitude, without changing any action. This directly demonstrates why TrajAuditor's PPR cannot serve as stand-alone evidence of behavioral unlearning.

  • The retraining reference accounts for variation between training runs through independent reference and held-out policies. The larger experiments use 50 reference retrains and 40 held-out retrains, with 40/40 and 39/40 held-out policies accepted in the informative HalfCheetah settings. These checks provide useful evidence that the audit recognizes genuine retraining outcomes.

Critical weaknesses
  • The strongest failure evidence comes from two informative settings, both involving 30% deletions in HalfCheetah. As the authors acknowledge, the larger Hopper deletions remain uninformative under this audit, so broader applicability is still somewhat uncertain. Another informative environment or independently selected deletion masks could help clarify how consistently the findings extend beyond the current settings.
Questions for authors

N/A

1 comment

Response 1: Additional tasks and deletion masks fixed in advance

Thank you for suggesting another environment and independently selected deletion masks. We have added deletions that the audit detects in Walker2d and PointMaze medium, each with 50 reference retrains, and in both the audit accepts 40/40 independent held-out retrains.

In Walker2d, for an ordered deletion of 45% of dataset episodes, the audit rejects 3/3 unchanged policies and 3/3 released TrajDeleter outputs; one released output passes the return band but fails occupancy (Table H.3). Calibration, visibility and the rejection of the released method persist across 33/33 measurement configurations. For a random request deleting the same number of episodes, the audit accepts 38/40 held-out retrains and rejects 0/3 unchanged policies. We chose the 45% rate after earlier 30% Walker2d requests were uninformative; both 45% masks were then fixed before fitting, and we report the random one as a negative control because it shows that visibility depends on the deletion request (Table H.4; Appendix H.3).

PointMaze adds sparse-reward navigation, where the request deletes the union of 3 fixed route clusters. With navigation goals in the occupancy representation, the audit rejects 3/3 unchanged policies and 3/3 IQL reward-negation ports, although the port keeps a success rate of 0.851–0.921. Retained-only continuation at the full budget passes in 3/3. Across 3 reset schedules, held-out acceptance stays at 40/40, the unchanged policies are rejected in 3/3, and the port is rejected in 2/3, 3/3 and 3/3. Occupancy pooled without goal labels accepts the edited policies while native return separates them; Tables H.1–H.2 report both measurements.

Walker2d uses the released procedure, while navigation uses our documented port of reward negation. We also tried D4RL Kitchen as a manipulation task, but its policies failed the fixed utility screens and give no manipulation evidence (Appendix H.3). The additions extend the evidence to a second locomotion task and to sparse-reward navigation.

Reviewer_rvEWOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates trajectory unlearning in offline RL and argues that unlearning should be evaluated by comparison to policies retrained without the deleted data. The authors introduce a retraining-calibrated behavioral audit with an unchanged-policy visibility check, analyze why suppression-based unlearning objectives can differ from true deletion, and evaluate several such methods on the TrajDeleter benchmark. They find that the benchmark's standard deletions are often too small to distinguish retraining from the unchanged policy, while on larger detectable deletions the evaluated suppression methods fail to reliably reproduce retraining behavior. The paper additionally demonstrates limitations of TrajAuditor's PPR forgetting score.

Critical strengths

A key strength of the paper is its careful separation of suppression from genuine deletion. The unchanged-policy visibility gate is particularly valuable: it prevents an unlearning method from receiving credit in settings where the original policy is already indistinguishable from retraining. The combination of a retraining-calibrated behavioral audit, a clear per-state analysis of why negative weighting and reward negation need not recover the retraining target, and targeted empirical mechanism checks makes the critique technically convincing rather than merely empirical. I also found the constant critic-shift experiment especially compelling, as it provides a simple demonstration that the existing PPR score can indicate complete forgetting without any change in policy behavior. Overall, the paper identifies important shortcomings in current offline-RL unlearning methodology and provides a principled framework for evaluating future methods.

Critical weaknesses

The main issue I see: The distinction from TOUR needs to be made clearer. Since TOUR already provides criteria for assessing successful offline-RL unlearning, it is currently unclear what the proposed audit captures that TOUR's evaluation does not. A direct empirical comparison on the same methods/settings seems important to establish the added value of the proposed audit. In particular, Figure 4 demonstrates that PPR is unreliable, but does not compare against TOUR's evaluation criteria. I also believe other central insights from this paper are already present in TOUR.

Other than that I found a couple issues worth addressing, but these are comparatively minor:

A) I personally cannot really judge how relevant the addressed scenario is in real world settings. Even though I can somehow imagine the situation, I would appreciate it if the authors could add a small discussion / examples, where all 3 conditions necessating their approach are present:

  1. individual contributors or trajectory groups can request deletion
  2. full retraining is genuinely expensive
  3. removing those trajectories produces a meaningful behavioral change

B) How sensitive are the visibility conclusions in Figure 3 to the choice of the 257-cell partition? In particular, could the benchmark and Hopper deletions become detectable under a finer partition or a continuous two-sample statistic, given sufficient rollouts?

C) The paper motivates state-action occupancy distance as a stronger notion of behavioral fidelity than return alone, which I find reasonable. However, I would appreciate some discussion of the semantic meaning of this distance. The metric treats differences in occupancy uniformly, although state-action differences can vary substantially in their behavioral relevance: a small occupancy change in a critical part of the state space may have a large effect on task behavior or return, whereas a much larger change in regions where actions are largely interchangeable may be practically inconsequential. To what extent should a given occupancy distance therefore be interpreted as a meaningful difference in policy behavior, beyond statistical distinguishability? Could complementary task-dependent metrics help characterize this distinction?

D) In Fig. 4(b), why are the critic-raising controls only shown for a subset of the larger shift magnitudes, rather than over the same range as the critic-lowering experiment? It would be helpful to clarify whether the full positive-shift sweep was evaluated and, if so, to show it for symmetry.

E) While the experimental methodology is carefully described in the text, I found several figures difficult to interpret without repeatedly cross-referencing the surrounding sections and appendices (this might be a failure on my end however). For example, understanding Figure 3 requires the reader to infer that the blue points are held-out retrains rather than the reference retrains defining the family radius, that the reference retrains themselves are a third set and not shown, and that the pooled benchmark column contains separately normalized settings. Similarly, terms such as "retained" vs "retrained, "family", “unchanged,” “original,” and “released” are not always sufficiently self-contained. Figure 4 is also somewhat difficult to parse because both panels share a single legend despite using different subsets of its entries and different visual encodings. I encourage the authors to make the figures more self-contained, define these categories directly in the captions, and provide separate legends for the two panels of Figure 4.

F) For table 1, I would appreaciate if you could add a row for the held-out retrains.

Questions for authors

see above

2 comments

Response 1: Comparison with TOUR

Thank you for asking for this comparison; you are right that Figure 4's critique of PPR does not by itself show what our audit adds to TOUR. We have added a direct comparison and clarified the attribution. TOUR already uses retraining references, matched non-members, retained utility and multiple attacks, and it reports audit sensitivity and the privacy–utility failures of GA. The revision credits these findings to TOUR.

Our addition is a retraining-calibrated test of closed-loop visitation, the states and actions a policy visits when it acts, together with the unchanged-policy gate and its tight visibility bound. Return measures one reward function, and membership attacks measure the information available through their queries; neither determines the visitation distribution. TOUR's attack-power control asks whether the original model carries a membership signal, whereas our visibility gate asks whether the original policy visits states and actions differently from retraining. On the released Decision Transformers (DTs), the originals carry a membership signal in 3/3 cases while occupancy rejects 0/3, so the two controls test different hypotheses.

For the informative HalfCheetah IQL 30% deletion, we applied TOUR's released likelihood evaluators to all 105 existing checkpoints. The additional original-policy seeds give:

CohortPrimary TOST5 point gapsRetained-NLL shiftOccupancy rejected
Unchanged0/76/77/77/7
GA+Refit7/77/77/77/7
Held-out retrains40/4040/4040/401/40

The first three columns are separate diagnostics. TOST, the primary equivalence test, uses margin 0.1; the point-gap checks and the 20% shift rule for retained NLL add evidence but give no joint equivalence guarantee. At 200 rollouts, the GA+Refit return relative to retraining is 0.940 (90% bootstrap interval over policies and resets [0.920, 0.959]): within our 10% tolerance but outside 5%, and the retained AUCs differ. The matching and head choices adapt TOUR to IQL, and our GA refits the policy variance, which TOUR keeps frozen. The table therefore shows disagreement between individual diagnostics and occupancy, short of a pass of TOUR's full privacy and utility evaluation (Table I.2).

We also ran the official DT code: 162 full-budget phases over 9 settings, 3 seeds and both fixed GA weights. Our runs reproduce the published TOST and point-gap decisions in 15/15 published conditions, but numerical reproduction is partial: 16/45 conditions meet all 4 fixed checks, and Appendix I.3 reports the discrepancies. The pinned release does not include the representation and shadow attacks, so our implementations of those are adaptations.

We then audited the released DT setting itself: its 10% training-episode deletion, with 50 reference retrains and 111 policy measurements. Occupancy accepts 40/40 held-out retrains and all originals, so an edited policy's acceptance cannot demonstrate behavioral deletion here. Both GA variants fail occupancy and the return band. Fine-tuning meets the point-gap and shift checks in 3/3 and the return band in all cases, but occupancy rejects 1 of the 3 policies, and primary TOST passes 0/3. The 40 fresh held-out retrains pass TOST in 0/40 but the point-gap and shift checks in 40/40, the same pattern that TOUR reports for its retrained (Gold) models. A TOST failure therefore does not by itself indicate residual membership. These results use the release's normalization, computed on the full training data (Table I.5; Appendix I.4).

The revision therefore presents occupancy as behavioral evidence that complements TOUR's information and utility measurements. We have not found a policy that passes TOUR's entire evaluation while failing occupancy, and we make no claim of overall superiority.

Response 2: Practical scope, sensitivity and presentation

A. A practical setting with all three conditions

The Introduction now opens with such a setting. Crowdsourced trajectories can be grouped by contributor or site, and a contributor may withdraw consent or ask to remove trajectories that contain personal information (condition 1). Retraining a large policy after every request can be expensive (condition 2). Whether removing a group changes the learned decisions, for example when it contains distinctive maneuvers, is exactly what the visibility gate checks (condition 3). PointMaze provides controlled evidence for the third condition through a route deletion; validation in a deployed system remains open.

B. Sensitivity of Figure 3

At the benchmark's 1%–5% deletions, calibrated detections occur only for Hopper TD3+BC 5% on development models: with the kernel estimator, and with 512- and 1024-cell partitions at 10 rollouts. These partitions lose calibration at larger budgets, and fresh models did not confirm the kernel detection. HalfCheetah 30% stays visible and Hopper 45% stays undetected. We therefore report a non-detection as a property of the tested measurement and do not conclude that the deletion has no behavioral effect.

In Walker2d, the ordered deletion keeps calibration, visibility and the rejection of the released method in 33/33 configurations, while the random deletion calibrates in 31/33 without becoming visible. Appendices G–H report the configurations, calibration failures and changes in method verdicts.

C. What an occupancy difference means

We agree that equal amounts of occupancy change need not matter equally for the task. A rejection establishes that a policy is incompatible with the observed retrain family at the measured resolution; whether the difference causes practical harm is a separate, task-dependent question. Likewise, a binned acceptance need not imply agreement on every behavior or temporal property.

We therefore report return and success alongside occupancy. In PointMaze, high goal-wise success coexists with occupancy rejection, and native return also separates the edited policies. Prespecified metrics for critical events or region visits could show which differences matter for a given task; Appendix A.3 now discusses this distinction.

D. The critic-raising sweep

The original raising control covered only part of the range. Figure 4(b) now shows the full matched sweep in both directions, at 0.001, 0.003, 0.01, 0.03, 0.1 and 0.3 of the original mean critic value on deleted rows, for the same 3 HalfCheetah TD3+BC actors. In both directions PPR reaches zero at 0.01, and no action changes. The raising sweep covers these 3 actors; the lowering result over all 39 evaluations is unchanged.

E. Captions, legends and terminology

We agree that the figures should be readable on their own. The Figure 3 caption now identifies the blue points as held-out retrains, states that a separate, unplotted set of reference retrains sets each radius and that independent pilots fit the partition, gives the sample sizes, and explains that the pooled benchmark settings are each normalized by their own radius. Figure 4 has a separate legend for each panel and defines its cohorts and encodings. The captions distinguish retained data, retrained (reference and held-out) policies, original policies and released edited outputs.

F. Held-out retrains in Table 1

Added. For HalfCheetah TD3+BC, HalfCheetah IQL and Hopper TD3+BC at 30% deletion, the held-out retrains have mean normalized distances 0.7, 0.8 and 0.6, returns 48, 37 and 112, and rejections 0/40, 1/40 and 0/40. These are the independent held-out policies; the reference retrains that define the radius are not part of this row.

Reviewer_pNimOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether existing offline RL unlearning methods genuinely reproduce retraining behavior or merely suppress deleted trajectories. The authors propose a retraining-calibrated behavioral audit based on state–action occupancy, incorporating multiple independent retrains and an unchanged-policy control. The main limitation is that informative experimental evidence is largely restricted to HalfCheetah.

Critical strengths
  1. The paper introduces an unchanged-policy visibility gate, which checks whether the original policy is distinguishable from retraining before assessing unlearning effectiveness. This prevents methods from receiving credit when deletion has no detectable behavioral effect. Combined with an occupancy-based behavioral audit calibrated against a family of independent retrains and a tight visibility bound based on Le Cam's argument, this provides a well-motivated evaluation framework.

  2. Building on prior findings about negative-weight objectives in supervised unlearning, the paper develops a per-state actor–critic analysis for offline RL. It explains how negative weighting, balanced sampling, and reward negation can cause suppression objectives to deviate from retraining. The analysis provides testable predictions, which are largely supported by experiments on HalfCheetah, although some predicted effects are weaker or absent in other settings.

Critical weaknesses
  1. The evidence that suppression methods fail on behaviorally visible deletions comes primarily from HalfCheetah with a 30% deletion rate. Hopper remained uninformative even at 45% deletion, while calibration difficulties arose in other settings. It remains unclear whether these findings generalize to discrete control, sparse-reward navigation, or robotic manipulation tasks.

  2. The proposed audit relies on a fixed 257-cell state–action partition and 50 rollouts per policy. Its effectiveness may depend on partition resolution and rollout count, yet the paper lacks a systematic sensitivity analysis demonstrating whether the acceptance decisions remain consistent across these choices. Furthermore, extending this occupancy-based approach to high-dimensional or vision-based environments may be challenging.

  3. Although the introduction motivates unlearning through scenarios such as consent withdrawal, the proposed audit evaluates only behavioral agreement with retraining. A policy could pass the audit while still retaining information about deleted trajectories in its parameters or critic. While the authors acknowledge this limitation, the absence of complementary information-level evaluations leaves the relationship between behavioral fidelity and actual information removal unexplored.

Questions for authors
  1. How sensitive are the audit’s conclusions on the benchmark’s 1% and 5% deletion settings to partition resolution (e.g., 512 or 1024 cells), the number of rollouts, or alternative occupancy estimators? In particular, would the unchanged policy become distinguishable from retraining at higher resolution? This would clarify whether these deletions have negligible behavioral effects or are simply below the audit’s current detection capability.
  2. A 30% deletion is behaviorally visible on HalfCheetah, whereas a 45% deletion remains undetectable on Hopper. Can the authors characterize what drives this difference, such as trajectory redundancy, state–action coverage, or the distribution of local deleted shares ρ(s, a)? Understanding these factors could help predict when the proposed audit will be informative.
  3. Beyond the targeted Hopper deletion in Table D.7, have the authors investigated targeted deletions on HalfCheetah, such as removing trajectories associated with specific behaviors or regions of the state space? Could such deletions produce detectable behavioral changes at smaller deletion rates (e.g., 1%–5%), where random deletions remain indistinguishable from retraining?
  4. On the two informative HalfCheetah settings, would information-level evaluations, such as TOUR’s membership-inference attacks, produce conclusions consistent with the proposed behavioral audit? In particular, how should cases be interpreted where a method passes an information-level test but fails the occupancy audit, or vice versa? Clarifying this would help establish which aspects of unlearning the framework can reliably assess.
2 comments

Response 1: Empirical breadth and measurement sensitivity

Thank you for identifying the limits of the original evidence. We have added two tasks in which the deletion is detected and a systematic sensitivity study, and we report the negative results alongside them.

Evidence beyond HalfCheetah

In both new tasks the audit uses 50 reference retrains and accepts 40/40 independent held-out retrains. In Walker2d, for an ordered deletion of 45% of dataset episodes, the audit rejects 3/3 unchanged policies and 3/3 released TrajDeleter outputs; one of these outputs passes the return band but fails occupancy. For a random deletion of the same number of episodes (a smaller share of transitions, 0.499 against 0.621), the audit accepts 38/40 held-out retrains but rejects 0/3 unchanged policies, so that request is not detected. We chose the 45% rate after earlier 30% Walker2d requests were uninformative; both 45% masks were then fixed before fitting, and the contrast between them shows why visibility must be checked for each request (Tables H.3–H.4).

In sparse-reward PointMaze medium, the request deletes the union of 3 fixed route clusters. With navigation goals included in the occupancy representation, the audit rejects 3/3 unchanged policies and 3/3 IQL reward-negation ports, although the port keeps a success rate of 0.851–0.921 (retrain median 0.8125; fixed floor 0.8). Retained-only continuation at the full budget passes in 3/3. Across 3 reset schedules, held-out acceptance stays at 40/40 and unchanged-policy rejection at 3/3, and the port is rejected in 2/3, 3/3 and 3/3. Occupancy pooled without goal labels accepts the edited policies, while native return still separates them; Tables H.1–H.2 report both.

Walker2d uses the released TrajDeleter procedure; for navigation we evaluate our own documented port of reward negation. We also tried D4RL Kitchen as a manipulation task, but both cohorts (12 models in total) failed the fixed utility screen and therefore give no evidence either way. Generality to discrete control and manipulation remains open (Appendix H.3).

Q1. Resolution, rollout count and alternative estimators

We tested partitions with 128, 256, 512 and 1024 non-absorbing cells (plus an absorbing cell), Gaussian-kernel features at 3 bandwidths set on pilot policies, and rollout budgets from 10 to 200. At the benchmark's 1%–5% deletions, only Hopper TD3+BC 5% shows calibrated detections (rejection of at least 2 of 3 unchanged policies), and only on development models: with the kernel estimator at several bandwidths and budgets, and with the 512- and 1024-cell partitions at 10 rollouts. At larger budgets these finer partitions fail calibration (at 200 rollouts they reject 2 of 3 unchanged policies but accept only 8/10 held-out retrains), and the kernel detection was not confirmed on fresh models (below). No other benchmark setting becomes visible under any tested measurement. The remaining deletions may still have effects below the power of the tested measurements, so we do not conclude that they are behaviorally negligible.

The two HalfCheetah 30% deletions stay visible at 50, 100 and 200 rollouts, with all held-out retrains accepted at 200, and Hopper 45% stays undetected. For the Hopper 5% kernel detection (median bandwidth: 9/10 held-out retrains accepted, 2/3 unchanged policies rejected on development models), a confirmation with fresh models gave 36/40 and 0/3.

Some individual method verdicts change with the measurement. Across usable benchmark comparisons, the released TrajDeleter output changes from accept to reject in 4/759, and the HalfCheetah reward-hinge port changes from reject to accept in 58/192. Tables G.1–G.4 report the sensitivity results and calibration failures. In Walker2d, the ordered deletion keeps calibration, visibility and the rejection of the released method in 33/33 configurations, while the random deletion calibrates in 31/33 without becoming visible (Appendix H.3).

All our tasks have low-dimensional observations. A vision-based audit would need a separately fitted representation and its own calibration, and its detection power is untested (Appendix A.3).

Response 2: Deletion structure, targeting and information-level tests

Q2. Why HalfCheetah 30% differs from Hopper 45%

The difference lies in where the data are removed and in how much retraining varies, more than in the percentage of episodes. In the TD3+BC diagnostic, the HalfCheetah request removes a fraction 0.334 of the training transitions, against 0.129 for Hopper. Weighted by where the original policies go (256 diagnostic cells), the local deleted share is 0.665–0.699 for HalfCheetah and 0.070–0.076 for Hopper. The Hopper request therefore removes much less data from the regions these policies visit, and Hopper's retrain family is also more spread out (radius 0.998, against 0.311 for HalfCheetah).

Hopper does lose all retained data in some cells, but its original policies place no measured occupancy there. Loss of coverage alone therefore does not explain the difference, and the data do not support calling Hopper simply more redundant. Appendix E reports all 4 diagnostic resolutions. These are descriptive findings on development data; we have not validated them as a predictor of visibility.

Q3. Targeted HalfCheetah deletions at 1%–5%

Yes. We deleted the top-ranked episodes by return and, separately, by maximum forward velocity, at both deletion rates, with the rankings fixed in advance. Each setting uses 11 reference retrains, 10 held-out retrains and 50 rollouts:

Mask1%: held-outs accepted; originals rejected5%: held-outs accepted; originals rejected
Top return9/10; 0/39/10; 0/3
Top velocity10/10; 0/39/10; 0/3

All 4 settings calibrate, but in none is an unchanged policy rejected. Partition and kernel measurements at 10–50 rollouts also accept all unchanged policies wherever calibration passes (Appendix F).

We agree that targeting distinctive behavior is a sensible way to look for informative small deletions. Our rankings remove whole episodes and do not isolate a state–action region, so small localized requests remain an open question. The result also shows why the unchanged policy must be checked first: a request chosen to be distinctive need not change the learned policy beyond retraining variation.

Q4. Membership tests and disagreement with occupancy

We now compare membership and occupancy measurements on the same checkpoints in both informative HalfCheetah settings. For IQL, GA+Refit policies from 7 additional original-policy seeds meet TOUR's released primary equivalence test (TOST), its 5 point-gap checks and its retained negative log-likelihood (NLL) shift check, yet occupancy rejects 7/7. The same likelihood diagnostics accept 40/40 held-out retrains, and occupancy accepts 39/40. Table I.2 reports all 105 checkpoints.

At 200 rollouts, the GA+Refit return relative to retraining is 0.940 (90% bootstrap interval over policies and resets [0.920, 0.959]). This is within a 10% utility tolerance but outside 5%, and the retained-data AUCs also differ, so these IQL adaptations do not amount to a pass of TOUR's full privacy and utility evaluation.

The reverse occurs in TD3+BC: 3 retained-only continuations trained for 100k updates pass occupancy but fail the adapted action-error criteria, which 37/40 held-out retrains meet. We read these disagreements as evidence that the two kinds of test measure different targets. Membership agreement does not establish retraining's visitation, and occupancy acceptance does not certify that information about the deleted data has left the parameters or the critic. The revised paper states this scope (Section 5.6; Appendix I).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviewers

We thank the reviewers for their careful reading. The revised manuscript adds calibrated deletions that the audit detects in Walker2d and PointMaze medium (Section 5.5; Appendix H), comparisons with TOUR's membership tests on the same checkpoints (Section 5.6; Appendix I), and studies of deletion structure, measurement sensitivity and small targeted deletions (Appendices E–G). Figures 3 and 4 now have self-contained captions, Figure 4 has a separate legend for each panel and shows critic shifts in both directions, and Table 1 includes the held-out retrains. We also clarify what occupancy measures alongside information-level tests and where the evidence stops.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.