acceptodds
Under review as a conference paper at ICLR 2027

Replay What Matters: Off-Policy Replay for Efficient LLM Reinforcement Unlearning

11 views

Abstract

LLM unlearning has emerged as a cost-effective alternative to full retraining for removing hazardous knowledge from pretrained models while preserving general utility. Recent RL-based methods such as RULE reformulate unlearning as learning a refusal behavior, but their on-policy optimization repeatedly samples from the same forget and retain/boundary prompts throughout training. We identify a critical inefficiency in this process: easy cases quickly converge and provide little useful gradient signal, while hard cases near the forget/retain boundary continue to produce low-reward rollouts that are discarded after a single use. To address this issue, we propose ReRULE, an off-policy replay enhancement for reinforcement unlearning. ReRULE stores low-reward hard-case rollout groups in a replay buffer during early GRPO training and reuses them in later stages through importance-sampled off-policy updates, redirecting computation toward boundary cases that require further learning. Theoretically, we show that ReRULE yields a tighter hard-case convergence bound than pure on-policy RULE. Empirically, ReRULE improves MUSE-Books Retain Quality from 46.3 to 56.2 while adding only 5–11% training time across benchmarks. Its limited improvement on the simpler TOFU setting further supports the intended conditional behavior: replay is most beneficial when the hard/easy disparity is pronounced.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Unlearning, Reinforcement Learning, Responsible AI

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_KG6aOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this submission, the authors propose Replay-enhanced Reinforcement UnLEarning (ReRULE), an off-policy replay mechanism. ReRULE performs hybrid training: it keeps the standard on-policy GRPO update while adding an importance-sampled off-policy update over replayed hard cases. The main contributions include the observation of training inefficiency in RL-based LLM unlearning, the ReRULE framework iteself, and related validation experimental results.

Critical strengths
  1. Clear motivation. The paper identifies an important inefficiency problem in on-policy reinforcement unlearning and proposes a simple yet practical solution. The paper is well structured and easy to follow.

  2. Impressive results. The experiments show promising improvements on MUSE-Books, and the random-replay ablation supports the importance of proposed design.

Critical weaknesses
  1. Questionable theoretical justification.

In Appendix E, the authors treat replayed trajectories as additional effective samples when deriving the generalization bound in Eq. (30), without adequately accounting for their statistical dependence. Moreover, comparing upper bounds does not directly justify the strict improvement claimed in Eq. (32).

  1. Insufficient evidence for training efficiency claim.

ReRULE retains the original on-policy updates while introducing additional off-policy updates, thus increasing the total training time (Table 7). Due to the lack of computationally matched comparisons or time-to-quality evaluations, it remains unclear whether the improvements result from more efficient optimization or simply additional computation.

  1. Limited methodological novelty.

ReRULE primarily combines established techniques, including reward-based hard-case selection, experience replay, and importance-weighted off-policy updates. While their application to reinforcement unlearning is useful, the methodological novelty beyond adapting existing RL techniques appears limited.

Questions for authors
  1. Could the authors clarify how the statistical dependence of replayed trajectories is handled in Eq. (30), and what assumptions justify the risk improvement claimed in Eq. (32)?

  2. Could the authors provide compute-matched comparisons with RULE to better demonstrate the efficiency benefits of ReRULE?

2 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer KG6a

We thank you for your time and valuable feedback. Below is our response to your comments.

Weakness 1. Questionable theoretical justification

We thank the reviewer for pointing this out. Please refer to Point 1 of our “Response to Common Concerns Raised by Reviewers.”


Weakness 2. Insufficient evidence for the training efficiency claim

We discuss this in Figure 3. Figure 3(a) shows that ReRULE achieves better performance in fewer training steps than RULE. Figure 3(b) tracks different types of hard examples and shows that ReRULE enables the model to learn these examples in fewer training steps. We acknowledge that total training time increases, but consider the measured overhead acceptable given the performance gains.


Weakness 3. Limited methodological novelty

ReRULE is designed to address challenges specific to unlearning during RULE’s [1] reinforcement learning training for LLM unlearning. Its replay mechanism is also designed to be readily adaptable to future RL-based LLM unlearning methods.


Question

Please refer to our responses in the Weakness section. We sincerely thank you for your time and effort throughout the review process!


We look forward to your response and would be happy to discuss any aspect of the paper with you.

[1] RULE: Reinforcement UnLEarning Achieves Forget–Retain Pareto Optimality, nips2025

Thank you for the rebuttal and revisions. I acknowlege the removal of the unsupported theoretical claims and the clearer positioning of the method. With the theoretical contribution withdrawn, now I assess the paper primarily on its methodological and empirical merits, and I remain unconvinced that the current evidence sufficiently establishes the claimed advantages.

My concern about computational efficiency remains: matching training steps does not match computation when ReRULE adds replay updates. The random-replay ablation supports hard-case selection, but does not establish an advantage over RULE under the same computational budget.

I also share Reviewer 3UW2’s concern about checkpoint selection. Please specify the selection criterion, data split, and reported step for each method, and reconcile the RULE result in Table 2 with its apparent peak in Figure 3. The selection protocol and its consistent application need to be clear.

Reviewer_3UW2OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

ReRULE extends RULE, which combines refusal SFT with GRPO over forget and boundary prompts. The paper observes that forget prompts become easy after SFT, while boundary prompts remain difficult. ReRULE stores rollout groups with mean reward below a threshold in a FIFO buffer and adds an importance-weighted replay update after a warm-up period. The evaluation covers RWKU, MUSE-Books, and TOFU, with random-storage and threshold ablations, training-time measurements, and a theoretical argument in Appendix E.

Critical strengths
  1. The problem framing is clear: Fig. 1a shows that after refusal SFT, the bottleneck of RULE's RL stage is undoing over-refusal on boundary prompts.
Critical weaknesses

1. The experiments do not support the central claim. If the mechanism works as described, the benefit should appear as improved retain quality. It does not, except on MUSE-Books, and that one number is hard to interpret.

  • RWKU (Table 1). Retain quality is unchanged (73.7 →\to 74.1) and forget naturalness drops (89.1 →\to 82.1). The forgetting gain cannot be attributed to replay: the RULE row is the RULE paper's 20-step run, while ReRULE is trained for 37 steps (App. D).
  • TOFU (Table 5). All metrics are equal or slightly lower.
  • MUSE-Books (Table 2 vs. Fig. 3a). ReRULE is above RULE for much of training, but its retain quality peaks near 56 around step 172 and ends near 34, below RULE's final value; no checkpoint rule is stated. The RULE baseline (Utility 46.3, Helpfulness 38.2) is also well below the RULE paper's figures for the same setting (55.6 / 81.4, Table 10).
  • The curves show earlier progress on a few hard cases (Fig. 3b), not faster convergence or more stable retention at the benchmark level.

2. The method does not address the inefficiency the paper identifies, and its effect is not isolated.

  • Alg. 1 keeps ordinary on-policy sampling and adds an update, so training time increases on all three benchmarks (Table 7). No curve against wall-clock shows whether ReRULE reaches comparable quality sooner.
  • The random-storage control (Table 4) shows that storing arbitrary groups does not help, but it does not separate replay from spending the same compute on extra updates, or on fresh rollouts for hard prompts (DAPO-style re-sampling, cited in Sec. 2 but not compared).
  • The gap between τ=1.0\tau = 1.0 (Table 8, 53.7) and random storage (Table 4, 46.3) needs explanation, taking their different forgetting scores into account.
Questions for authors

Clarification requests:

  1. Which training step is reported in Tables 2, 4 and 8, and how was it selected for each method? Fig. 3a shows ReRULE's retain quality peaking near 56 and ending near 34; if Fig. 3a is computed on a different split from Table 2, please say so.
  2. Where do the baseline numbers come from? On MUSE-Books your RULE run is well below the RULE paper's values for the same setting (Utility 46.3 vs. 55.6); on RWKU the RULE row matches the RULE paper's 20-step run while ReRULE is trained for 37 steps, and the RULE-PPO and RULE-RPP variants are omitted. Were training budgets matched in the reported comparisons?
  3. Does ReRULE reach a comparable forgetting–retention trade-off sooner, or a better trade-off at the same training time? Neither is shown at present.
  4. Please define Normalize(·) and the clip range, and state whether replayed advantages are recomputed and how the on-policy and off-policy updates (Eq. 3 and Eq. 5) are combined per step. Appendix D does not cover these.
  5. What fraction of replayed groups are boundary prompts rather than forget prompts, and does the hard/easy split hold per prompt or only between the two categories (Fig. 1a shows category means)? If replayed groups are almost all boundary prompts, would increasing the boundary sampling ratio in RULE have the same effect?
  6. What causes the Helpfulness drop on RWKU (71.9 → 56.3)? Is it a side effect of pushing non-refusal on boundary prompts?

Suggestions:

  1. Add RULE with a matched number of gradient updates and a DAPO-style re-sampling baseline; this is the comparison that would show replay, rather than extra updates on boundary prompts, causes the gain.
  2. The related-work comparison to DQN-era replay does not establish what is new. Please discuss how ReRULE relates to ExGRPO (ICLR 2026), which replays selected GRPO rollouts with importance weighting, and to PURGE (arXiv 2601.20568), a GRPO-based unlearning method evaluated on RWKU.
  3. Appendix E is labelled a theorem but has no proof, counts replayed rollouts as new samples, leaves the importance-sampling bias term undefined, and infers a risk ordering from a comparison of upper bounds. Either provide the proof or present it as an informal argument and narrow the claim in the abstract.
  4. Smaller fixes: the threshold, minimum buffer size and warm-up step differ per benchmark without a selection procedure, and sensitivity is shown only on MUSE (τ = 0.6 breaks forgetting, Table 8); Appendix C.1 defines FB and AA differently from Sec. 4.2; +GDR and +KLR are never defined; Table 1 highlighting is wrong in the AA and Help columns; equation numbers (24)–(26) are duplicated.
3 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer 3UW2

We thank you for your time and valuable feedback. Below is our response to your comments.

Weakness 1. The experiments do not support the central claim

We thank the reviewer for raising these experimental concerns. We first clarify that ReRULE selects low-reward trajectory groups from both the forget and retain sides. Its benefits may therefore appear as improvements in either forgetting or retention, rather than exclusively in retain quality.

  1. Differences in Helpfulness evaluation. We used a different external LLM for Helpfulness evaluation from that used in RULE [1]. The Helpfulness scores across the two papers are therefore not directly comparable, which also explains the substantial numerical discrepancy. RULE and ReRULE in our paper were evaluated under the same settings.

  2. Checkpoint selection on MUSE-Books. The reported results correspond to the checkpoints at which RULE and ReRULE respectively achieve their peak retain quality. ReRULE reaches a higher retain-quality peak in fewer training steps, consistent with our motivation of improving training by revisiting hard cases through experience replay. We acknowledge that this result reflects fewer steps to reach a better peak, rather than more stable retention at the end of training.

  3. Training steps on RWKU. To control for differences in training steps, we have added results for RULE trained for 37 steps and compared them with ReRULE trained for the same number of steps. The results are shown below:

MethodForget FB ↓Forget QA ↓Forget AA ↓Forget All ↓Retain FB ↑Retain QA ↑Retain All ↑
ReRULE (GRPO)26.513.133.224.279.568.874.1
RULE (37 steps)40.1822.6548.2737.0381.0266.8173.92

Weakness 2. The method does not address the inefficiency the paper identifies, and its effect is not isolated

  1. Efficiency. Although Table 7 shows increased training time across all three benchmarks, Figure 3 shows that ReRULE achieves better forgetting and retention performance in fewer training steps than RULE. Furthermore, Figure 3(b) tracks three hard examples and shows that ReRULE learns these examples substantially faster in terms of training steps than RULE. We have added an explanation of this point in Section 4.5 of the paper and <span style="color: blue;">highlighted changes in blue</span>.

  2. Source of the gains. The random-storage control retains the same additional updates and changes only the sample-selection strategy, supporting the importance of hard-case selection. However, it does not establish whether replay outperforms additional updates or fresh rollouts for hard prompts under the same compute budget.

  3. Difference between the two tables. The result of 53.7 in Table 8 uses a threshold of 1.0. Since the selection criterion is strictly less than 1.0, it still excludes groups with a mean reward of 1.0 and retains a difficulty-based filtering effect. This differs from the random-selection mechanism in Table 4.


Questions

Clarification requests

  1. Please refer to our responses in the Weakness 1.

  2. On RWKU, we used the results reported in the original RULE paper and provided results with matched training steps in the Weakness section. On MUSE-Books, since a fine-tuned checkpoint required for RULE’s experiments was not publicly available, we reran the experiments using its released code and faithfully reported our reproduced results, which differ from those in the original paper. Our MUSE-Books experiments used identical configurations for RULE and ReRULE, with experience replay being the only difference.

  3. Please refer to our responses in the Weakness 2.

  4. The normalization operation and clipping range are defined as follows: Normalize⁡(⋅)\operatorname{Normalize}(\cdot) applies softmax over all token-level ratios in the replay batch after clipping them to [0.5,2][0.5, 2]. This definition and the remaining implementation details have been included in the revised manuscript in section 3.3 and <span style="color: blue;">highlighted in blue</span>.

  5. Hard-case selection operates at the level of the trajectory group associated with each prompt: a group enters the replay buffer when its mean reward falls below the threshold. It does not label the entire forget or boundary category as hard or easy. ReRULE selects and reuses historical trajectories based on their observed rewards, whereas increasing the boundary sampling ratio generates more fresh trajectories from that category. Analyzing the proportions of different prompt categories among hard cases is beyond the scope of this work, but we consider it an interesting direction for future research.

  6. Please refer to our responses in the Weakness section.

Suggestions

  1. Please refer to our responses in the Weakness section.

  1. We thank the reviewer for recommending these two relevant works. Both ExGRPO and ReRULE use experience replay with GRPO, but their selection units and criteria differ. ExGRPO stores successful reasoning trajectories and selects experiences based on question difficulty and trajectory entropy. ReRULE targets reinforcement unlearning, selecting complete low-reward trajectory groups by their mean rewards and reusing their original within-group advantages. Our contribution therefore lies in selecting and utilizing hard trajectory groups for reinforcement unlearning, rather than GRPO replay itself. PURGE formulates unlearning as a verifiable GRPO task by penalizing mentions of forbidden concepts. ReRULE retains RULE’s reward and training framework and focuses on reusing under-utilized hard trajectories. The two methods primarily address unlearning objective design and training experience utilization, respectively. We have added a discussion of these methods to the related work section, highlighted in blue.

  1. We thank the reviewer for pointing this out. Please refer to Point 1 of our “Response to Common Concerns Raised by Reviewers.”

  1. Thank you for pointing these out! We have addressed these issues in the revised manuscript, with the changes highlighted in blue.

We look forward to your response and would be happy to discuss any aspect of the paper with you.

Thank you for the response and the revision. I have read the rebuttal and the revised manuscript. Some points are now clear, but the two main weaknesses remain, so I keep my rating.

Resolved

  • Implementation and related work. The implementation details (Q4) and the relation to ExGRPO and PURGE are now covered.
  • Presentation. +GDR/+KLR, the FB/AA definitions, the equation numbering and the Table 1 highlighting in the AA and Help columns are fixed.
  • τ = 1.0 vs. random storage. The explanation of the difference is reasonable. Please add it to the paper.

Follow-up needed

  1. Checkpoint selection (W1, Q1). I understand that Table 2 reports each method's peak retain quality. Please state in the paper how the checkpoint is selected for every row you ran (RULE, ReRULE, random storage, the τ variants), on which data, and at which step. If the selection uses the reported metric itself, final-step results are needed as well. In Fig. 3a the RULE curve appears to peak near 50, which does not match the 46.3 in Table 2, and ReRULE is lower at the end of training, which does not match "consistently achieves better Retain Quality" in Sec. 4.3.

  2. Baseline sources (Q2). RULE is rerun on MUSE-Books but taken from the RULE paper on RWKU. Please mark each row as rerun or cited, and compare ReRULE against RULE from your own setup on every benchmark (item 3 covers RWKU). On MUSE-Books the gap between your RULE and the published one (Utility 46.3 vs. 55.6) is about as large as the reported gain; please report both and note any known differences in setup.

  3. The 37-step result on RWKU (W1). Thank you for the additional run. At matched steps the result favours ReRULE (Forget All 24.2 vs. 37.0). However, RULE is worse on every forget metric at 37 steps than at 20 steps (Forget All 27.7), and the two numbers come from different sources, so it is unclear whether this is due to the step count or the setup. RULE at 20 steps in your setup (the step-20 checkpoint of your 37-step run would suffice) and the step at which each result is reported (Appendix D says "maximum of 37 steps") are needed. Please also add this result to Table 1.

  4. Helpfulness on RWKU (Q6). The rebuttal says that Helpfulness was scored by a different judge from the RULE paper's, that RULE and ReRULE were evaluated under the same settings, and that the RWKU RULE row is taken from the RULE paper. If the naturalness scores of that row come from the RULE paper's judge and ReRULE's from yours, the columns in Table 1 are not comparable and the drop from 71.9 to 56.3 may be due to the evaluation. If both used the same judge, the drop is real, and the paper's explanation (a sharper refusal boundary) needs support. Please clarify which is the case for RWKU.

  5. Efficiency (W2, Q3). I accept that the overhead is small and do not ask for wall-clock curves. However, the evidence appears to come from single runs with large fluctuations. In Case 1 of Fig. 3b the two curves already differ before replay starts at step 26, so the rebuttal's "substantially faster" is hard to support. As Reviewer 2y62 also noted, results over a few seeds would be needed, at least on MUSE-Books.

  6. Source of the gain (W2, S1). I agree that random storage controls for the number of extra updates, and I appreciate the acknowledgement that it does not establish whether replay is better than fresh rollouts on hard prompts at equal compute. That control exists only on MUSE-Books; on RWKU the 37-step RULE run matches steps, not gradient updates. The comparison with fresh rollouts on hard prompts (e.g., DAPO-style re-sampling) is still missing. This limitation should be stated in the paper.

Still open

  • Composition of the replay buffer (Q5). The response describes the difference in mechanism but declines to report the fraction of replayed groups from boundary versus forget prompts ("beyond the scope"). It is available directly from the buffer, and it is needed to support the statement that the benefit can appear on either side.
  • Hyperparameter selection. The response says the smaller fixes were addressed, but I could not find a selection procedure for τ, the minimum buffer size or the warm-up step in the revision. τ = 0.6, which raises VerbMem from 0.0 to 6.6 on MUSE-Books, is the value used on RWKU and TOFU.

My assessment is unchanged: the headline result depends on a peak-checkpoint comparison, and the effect of replay is not isolated. The decisive issues are items 1, 3 and 6.

Reviewer_2y62OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an off-policy replay enhancement method called ReRULE for reinforcement learning forgetting. Analysis of the RULE algorithm reveals a phenomenon where some cues are learned quickly, while others near the "reject/answer" boundary consistently perform poorly. Continuing to evenly distribute training resources in this situation may be inefficient. Therefore, this paper enhances the baseline algorithm through an engineering improvement of "hard sample selection + off-policy experience replay".

Critical strengths
  1. The authors addressed a real-world training resource allocation problem. Experiments demonstrated the difference in training rewards between forgotten and retained samples, providing an intuitive motivation for the potential computational waste associated with uniform sampling.

  2. Compared to RULE, ReRULE primarily adds a buffer, selection criteria, and additional updates in the second stage. The modification cost for existing RULE/GRPO is relatively manageable, resulting in a lightweight overall design.

  3. Regarding the reported metrics, ReRULE did not significantly sacrifice forgetting performance but improved utility performance (46.3 → 56.2).

  4. Effective ablation experiments were designed to demonstrate the effectiveness of hard samples, particularly regarding the crucial factor of replay buffer design.

Critical weaknesses

1. Insufficient Statistical Reliability and Generalization Range:

The paper does not report results, standard deviations, or confidence intervals for multiple randomized seeds. In particular, the 0.4-point difference in RWKU retention cannot be determined whether it exceeds random fluctuation.

Generalization range is also limited:

  • The main experiments are concentrated on the 7B/8B model;
  • TOFU only uses forget01;
  • Core ablation is concentrated on MUSE;
  • The effects of "difficulty ratio," "data volume," and "task type" were not separated through controlled experiments.

2. The theoretical derivation contains certain logical errors:

In general, the following inequalities: RA≤UAR_A \leq U_A, RB≤UBR_B \leq U_B, and UA<UBU_A < U_B do not imply that RA<RBR_A < R_B. However, Equations (27) and (32) make similar strict risk comparisons. Even if the preceding upper bounds hold, this conclusion cannot be derived from them alone.

Numerical counterexample. Let RA=0.3R_A = 0.3, UA=0.4U_A = 0.4, RB=0.1R_B = 0.1, and UB=0.5U_B = 0.5. Both upper bounds hold, and UA<UBU_A < U_B, yet the actual risks satisfy RA>RBR_A > R_B. To establish a strict ordering of the risks, one must directly compare the risks or provide additional lower bounds or structural arguments.

3. In terms of innovation, it's more like a hodgepodge of existing methods.

In terms of innovation, Prioritized Experience Replay studies the prioritized replay of important experiences; LUFFY studies large-scale mixed GRPO and importance sampling with the same/different strategies; D3S studies effective sample selection schemes. Papers should clearly distinguish:

  1. Existing ideas on replay, sample selection, and mixed strategies;

  2. Newly added applications of forgetting boundaries and trajectory group screening designs.

Questions for authors
  1. Corrected the risk comparison error in the theory: Refer to Weaknesses 2.

  2. Supplemented ablation experiments with multiple random seeds, multiple leaderboards, and multiple sizes: Refer to Weaknesses 1.

  3. Clarified the key design of advantage calculation: Should old advantages be preserved or recalculated? Are the actual generation strategy probabilities corresponding to each group preserved? Trajectory-level or token-level weights? Are advantages regrouped? These questions are not fully addressed in the main text, Algorithm 1, and Appendix D.

2 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer 2y62

We thank you for your time and valuable feedback. Below is our response to your comments.

Weakness 1. Limited statistical evidence and generalizability

Our experimental setup largely follows RULE [1] and FLAT [5]. Regarding model size, 7B and 8B models are commonly used in current LLM unlearning studies.


Weakness 2. The theoretical derivation contains certain logical errors

We thank the reviewer for pointing this out. Please refer to Point 1 of our “Response to Common Concerns Raised by Reviewers.”


Weakness 3. The contribution relative to existing methods needs clarification

ReRULE identifies the disparity between hard and easy samples in reinforcement unlearning and introduces group-level hard-case replay. Using RULE’s forget and retain rewards, it stores complete rollout groups with mean rewards below a threshold and reuses their trajectories and within-group advantages. This selects poorly performing groups on both sides, using mean reward as a measure of difficulty rather than distance to the forget/retain boundary.

Unlike PER [2], which prioritizes transitions by TD error, ReRULE selects complete groups by mean reward. Unlike LUFFY [3], which uses off-policy reasoning demonstrations, ReRULE reuses the model’s own low-reward rollouts. Unlike D3S [4], which selects samples and tokens using advantage variance and token-level information, ReRULE replays historical groups across training steps. We will discuss these methods in the related work section, with additions <span style="color: blue;">highlighted in blue</span>.

Our contribution is the integration and empirical validation of hard-case replay in reinforcement unlearning, rather than a new general-purpose threshold criterion. The random-storage ablation reduces performance to approximately the RULE level, supporting the importance of hard-case selection.


Question

1–2. Please refer to our responses above.

3.We thank the reviewer for pointing out that these implementation details were unclear. During replay, we reuse the advantages computed and stored after trajectory generation. We do not recompute advantages, regroup trajectories, or renormalize rewards. We also store the per-token log-probabilities under the sampling policy (old_log_probs). These values are computed after trajectory generation and before the policy update, and are stored in the buffer together with each trajectory group. The importance ratio is computed per token:

ρi,t=exp⁡[log⁡πθ(yi,t∣xi,yi,<t)−log⁡πold(yi,t∣xi,yi,<t)].\rho_{i,t} = \exp\left[\log \pi_{\theta}(y_{i,t}\mid x_i, y_{i, < t}) - \log \pi_{\mathrm{old}}(y_{i,t}\mid x_i, y_{i, < t})\right].

The processed weights are multiplied by the stored advantages token by token, rather than multiplying token probabilities across the entire trajectory to obtain a single trajectory-level weight.

We will clarify these details in Section 3.3 of the revised manuscript and mark the changes in blue.


We look forward to your response and would be happy to discuss any aspect of the paper with you.

[1] RULE: Reinforcement UnLEarning Achieves Forget–Retain Pareto Optimality, nips2025

[2] Prioritized Experience Replay

[3] Learning to Reason under Off-Policy Guidance, nips2025

[4] Learning More with Less: A Dynamic Dual-Level Down-Sampling Framework for Efficient Policy Optimization, ICLR2026

[5] LLM Unlearning via Loss Adjustment with Only Forget Data, ICLR2025


Thank you for the clarifications. Preserving complete rollout groups, their advantages, and the corresponding behavior-policy log-probabilities addresses several of my implementation questions. However, the justification of the historical replay update remains insufficient.

Fixed advantages and token-level ratios are not inherently incorrect, as they are used in standard GRPO. My concern is whether the update remains reliable as the current policy diverges from the policies that generated the replayed groups. GSPO discusses instability from token-level weighting, while μ-GRPO identifies prefix-support mismatch under high staleness. Moreover, clipping followed by batch-wide softmax produces heuristic rather than standard importance weights. The random-storage ablation supports hard-case selection, but does not establish the reliability of this weighting scheme. Policy-mismatch diagnostics and targeted weighting ablations would help address these concerns.

Overall, the rebuttal improves clarity but does not sufficiently resolve these methodological concerns. I therefore maintain my original score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to Common Concerns Raised by Reviewers

We thank all reviewers and the AC for their hard work and valuable feedback! Below, we provide a consolidated response to common concerns raised by multiple reviewers:

  1. Logical errors in Appendix E (Equations (27) and (32)). Upon checking, we confirm that these errors are indeed present, and we sincerely thank the reviewers for pointing them out. This analysis aimed to theoretically explain the advantages of ReRULE and RULE [1] over methods such as GA. We have revised the relevant parts (in Appendix and Section 1 Introduction) of the manuscript and will further refine this analysis in future work.

[1] RULE: Reinforcement UnLEarning Achieves Forget–Retain Pareto Optimality, NeurIPS 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.