- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper examines whether existing offline reinforcement learning unlearning procedures reproduce the behavior of training from scratch on the retained data. It proposes an audit based on discounted state–action occupancies, using independent retraining runs to define a reference family and an unchanged-policy control to check whether a deletion is detectable before interpreting an unlearning method's acceptance. On the TrajDeleter benchmark's 1% and 5% episode deletions, the unchanged policies usually pass wherever the retraining reference passes the authors' calibration check, making successful unlearning difficult to establish with this audit. For the informative 30% HalfCheetah deletions, the evaluated suppression procedures generally fail to match retraining, including additional runs of the released TrajDeleter procedure and a reward-hinge port. A per-state analysis attributes possible failures to negative actor weighting, amplification from balanced minibatches, and reward negation in the critic. The paper also demonstrates that constant shifts of a frozen policy's critic can drive TrajAuditor's forgetting score to zero without changing any action. The contribution is a critique of current evaluation practices and an accompanying behavioral audit.
Critical strengths
-
The unchanged-policy control exposes an important weakness in unlearning evaluation. In the calibrated benchmark settings, only one of 27 unchanged-policy evaluations is rejected. Checking this baseline first prevents acceptance of an edited policy from being mistaken for evidence of deletion when doing nothing would also pass.
-
The critic-shift experiment is particularly convincing. Keeping the actor fixed while shifting the critic makes all 39 tested evaluations report complete forgetting at the stated shift magnitude, without changing any action. This directly demonstrates why TrajAuditor's PPR cannot serve as stand-alone evidence of behavioral unlearning.
-
The retraining reference accounts for variation between training runs through independent reference and held-out policies. The larger experiments use 50 reference retrains and 40 held-out retrains, with 40/40 and 39/40 held-out policies accepted in the informative HalfCheetah settings. These checks provide useful evidence that the audit recognizes genuine retraining outcomes.
Critical weaknesses
- The strongest failure evidence comes from two informative settings, both involving 30% deletions in HalfCheetah. As the authors acknowledge, the larger Hopper deletions remain uninformative under this audit, so broader applicability is still somewhat uncertain. Another informative environment or independently selected deletion masks could help clarify how consistently the findings extend beyond the current settings.
Questions for authors
N/A
Response 1: Additional tasks and deletion masks fixed in advance
Thank you for suggesting another environment and independently selected deletion masks. We have added deletions that the audit detects in Walker2d and PointMaze medium, each with 50 reference retrains, and in both the audit accepts 40/40 independent held-out retrains.
In Walker2d, for an ordered deletion of 45% of dataset episodes, the audit rejects 3/3 unchanged policies and 3/3 released TrajDeleter outputs; one released output passes the return band but fails occupancy (Table H.3). Calibration, visibility and the rejection of the released method persist across 33/33 measurement configurations. For a random request deleting the same number of episodes, the audit accepts 38/40 held-out retrains and rejects 0/3 unchanged policies. We chose the 45% rate after earlier 30% Walker2d requests were uninformative; both 45% masks were then fixed before fitting, and we report the random one as a negative control because it shows that visibility depends on the deletion request (Table H.4; Appendix H.3).
PointMaze adds sparse-reward navigation, where the request deletes the union of 3 fixed route clusters. With navigation goals in the occupancy representation, the audit rejects 3/3 unchanged policies and 3/3 IQL reward-negation ports, although the port keeps a success rate of 0.851–0.921. Retained-only continuation at the full budget passes in 3/3. Across 3 reset schedules, held-out acceptance stays at 40/40, the unchanged policies are rejected in 3/3, and the port is rejected in 2/3, 3/3 and 3/3. Occupancy pooled without goal labels accepts the edited policies while native return separates them; Tables H.1–H.2 report both measurements.
Walker2d uses the released procedure, while navigation uses our documented port of reward negation. We also tried D4RL Kitchen as a manipulation task, but its policies failed the fixed utility screens and give no manipulation evidence (Appendix H.3). The additions extend the evidence to a second locomotion task and to sparse-reward navigation.