- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this submission, the authors propose Replay-enhanced Reinforcement UnLEarning (ReRULE), an off-policy replay mechanism. ReRULE performs hybrid training: it keeps the standard on-policy GRPO update while adding an importance-sampled off-policy update over replayed hard cases. The main contributions include the observation of training inefficiency in RL-based LLM unlearning, the ReRULE framework iteself, and related validation experimental results.
Critical strengths
-
Clear motivation. The paper identifies an important inefficiency problem in on-policy reinforcement unlearning and proposes a simple yet practical solution. The paper is well structured and easy to follow.
-
Impressive results. The experiments show promising improvements on MUSE-Books, and the random-replay ablation supports the importance of proposed design.
Critical weaknesses
- Questionable theoretical justification.
In Appendix E, the authors treat replayed trajectories as additional effective samples when deriving the generalization bound in Eq. (30), without adequately accounting for their statistical dependence. Moreover, comparing upper bounds does not directly justify the strict improvement claimed in Eq. (32).
- Insufficient evidence for training efficiency claim.
ReRULE retains the original on-policy updates while introducing additional off-policy updates, thus increasing the total training time (Table 7). Due to the lack of computationally matched comparisons or time-to-quality evaluations, it remains unclear whether the improvements result from more efficient optimization or simply additional computation.
- Limited methodological novelty.
ReRULE primarily combines established techniques, including reward-based hard-case selection, experience replay, and importance-weighted off-policy updates. While their application to reinforcement unlearning is useful, the methodological novelty beyond adapting existing RL techniques appears limited.
Questions for authors
-
Could the authors clarify how the statistical dependence of replayed trajectories is handled in Eq. (30), and what assumptions justify the risk improvement claimed in Eq. (32)?
-
Could the authors provide compute-matched comparisons with RULE to better demonstrate the efficiency benefits of ReRULE?
2 comments
Thank you for the rebuttal and revisions. I acknowlege the removal of the unsupported theoretical claims and the clearer positioning of the method. With the theoretical contribution withdrawn, now I assess the paper primarily on its methodological and empirical merits, and I remain unconvinced that the current evidence sufficiently establishes the claimed advantages.
My concern about computational efficiency remains: matching training steps does not match computation when ReRULE adds replay updates. The random-replay ablation supports hard-case selection, but does not establish an advantage over RULE under the same computational budget.
I also share Reviewer 3UW2’s concern about checkpoint selection. Please specify the selection criterion, data split, and reported step for each method, and reconcile the RULE result in Table 2 with its apparent peak in Figure 3. The selection protocol and its consistent application need to be clear.
Rebuttal to Reviewer KG6a
We thank you for your time and valuable feedback. Below is our response to your comments.
Weakness 1. Questionable theoretical justification
We thank the reviewer for pointing this out. Please refer to Point 1 of our “Response to Common Concerns Raised by Reviewers.”
Weakness 2. Insufficient evidence for the training efficiency claim
We discuss this in Figure 3. Figure 3(a) shows that ReRULE achieves better performance in fewer training steps than RULE. Figure 3(b) tracks different types of hard examples and shows that ReRULE enables the model to learn these examples in fewer training steps. We acknowledge that total training time increases, but consider the measured overhead acceptable given the performance gains.
Weakness 3. Limited methodological novelty
ReRULE is designed to address challenges specific to unlearning during RULE’s [1] reinforcement learning training for LLM unlearning. Its replay mechanism is also designed to be readily adaptable to future RL-based LLM unlearning methods.
Question
Please refer to our responses in the Weakness section. We sincerely thank you for your time and effort throughout the review process!
We look forward to your response and would be happy to discuss any aspect of the paper with you.
[1] RULE: Reinforcement UnLEarning Achieves Forget–Retain Pareto Optimality, nips2025