- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies trajectory unlearning: preventing an LLM agent from reproducing designated action sequences while preserving its ability to complete both target and other tasks. GiRPO augments policy rollout groups with penalized forget trajectories, isolates the reward-normalization statistics, and clips the negative forget advantages. Experiments on ALFWorld and WebShop show substantial reductions in trajectory repetition with comparatively well-preserved task success.
Critical strengths
GiRPO gives encouraging results on the stated sequence-avoidance task. On ALFWorld Clean, exact match falls from 33.9% to 1.4% while target success remains 88.4%, compared with 91.5% initially. On WebShop, target success improves while exact match falls.
Critical weaknesses
-
The practical motivation for forgetting the selected trajectories remains unclear. I found the example in lines 76–91 lengthy and difficult to connect to the actual evaluation. It introduces a poisoned trajectory with a covert action, but does not clearly establish why the entire trajectory is the necessary unit of removal. More importantly, the ALFWorld and WebShop experiments do not sufficiently explain what makes the selected trajectories undesirable or why a user would want them forgotten. The appendix provides trajectory examples, but these do not clearly connect a concrete unwanted behavior to its removal after unlearning ( the judge prompt in Figure 11 measures overall task-solving similarity rather than whether a particular unwanted behavior survives. ). This leaves me uncertain whether the evaluation captures a practical behavior-removal problem or primarily measures avoidance of particular recorded action sequences.
-
The baselines do not receive comparable opportunities to optimize the forgetting–success trade-off. GiRPO explicitly discourages forget trajectories while using online task rewards to learn successful alternatives on both target and untarget tasks. GA and NPO include retention losses, and DPO contrasts forget responses with substitute responses, but these objectives do not provide the same direct optimization of task success. GRPO, meanwhile, optimizes success only on untarget tasks without an explicit forgetting objective. NPO+GRPO is therefore the closest comparison, yet its GRPO component also excludes target tasks. The additional NPO v2 results partly address target-task coverage, but still do not match GiRPO’s online reward training. A comparison giving NPO+GRPO the same target and untarget tasks would help establish whether the reported advantage comes from GiRPO’s proposed group treatment rather than this difference in training opportunities.
Questions for authors
See weakness.
(1) What makes each designated trajectory undesirable, and which part must be removed? Can the existing rollout logs be evaluated for recurrence of that behavior or subtrajectory, including cases whose full sequence no longer matches? If the intended objective is strictly avoidance of recorded sequences, please clarify the scope of the safety claims accordingly.
(2) How does NPO+GRPO perform when its GRPO component receives the same target and untarget tasks, with a comparable rollout budget and the same exclusion of forget trajectories? This specific comparison would materially affect my assessment of GiRPO's contribution beyond access to alternative successful target-task behavior.
2 comments
Follow-up: Gentle Reminder Before the Discussion Deadline
Dear Reviewer bhLN,
With about 24 hours left in the discussion period, we would like to kindly follow up on our response.
For your concerns about the practical motivation (W1/Q1), we clarified why complete trajectories are the unit of an unlearning request, and added a poisoning experiment that connects a concrete behavior to its removal through a rule-based Behavior Recurrence Rate. For the baseline comparison (W2/Q2), we ran GA w/ GRPO (on both target and untarget tasks).
We would greatly appreciate your feedback on whether these additions address your concerns. Thank you very much for your valuable time and for helping us improve our work.
Response to Reviewer bhLN
We sincerely thank the reviewer for acknowledging the benefits of GiRPO and for offering insightful suggestions. Our responses are provided below.
Why complete trajectories are the unit of an unlearning request. Each individual action in an undesirable trajectory is typically reasonable by itself (e.g.,
run a microwave), and the problem lies in the way actions are combined or ordered under a specific task. Removing the individual action or behavior would restrict normal agent behavior. We therefore choose complete trajectories as the unit of an unlearning request, because an undesirable execution path may consist entirely of individually legitimate actions whose combination or ordering is unwanted. This formulation allows the agent to suppress a designated execution path while preserving the general availability of its constituent actions.What the selected trajectories are. In the ALFWorld and WebShop experiments, the forget trajectories are designated ones, as in unlearning benchmarks such as TOFU[1] and MUSE[2], where the forget set is designated rather than intrinsically harmful. They serve as a controlled testbed for whether a specific recorded trajectory can be removed while task utility is preserved. We do not claim these trajectories are intrinsically unsafe. To further illustrate our practical motivation, we have included an example of a poisoned trajectory in Figure 18 and its corresponding trajectory after unlearning in Figure 19.
Connecting a concrete behavior to its removal. To make this connection explicit, we added a poisoning experiment, where the undesirable behavior is putting an object into the garbagecan. Besides Exact Match, we measure the Behavior Recurrence Rate (BRR), the fraction of rollouts on the forget tasks that still exhibit the behavior. GiRPO reduces BRR from 92.5% to 62.9%, while NPO+GRPO barely changes it (88.5%) and NPO collapses. The LLM-as-Judge score differs from BRR by only a small margin (63.2% vs. 62.9%), so it also detects whether the behavior survives. The prompt and case studies are in Figs. 18-19.
*For NPO, the unlearned model collapses and outputs “take take …” at every step, so BRR = 0 does not mean the model has forgotten the behavior.
W2&Q2 The baselines do not receive comparable opportunities to optimize the forgetting–success trade-off.
We added the requested comparison, NPO w/ GRPO. In these baselines, the GRPO component optimizes online task rewards on both target and untarget tasks, with the same rollout budget and the same forget-trajectory exclusion as GiRPO. Results are on ALFWorld-Clean (Qwen2.5-3B-Instruct), shown in the table below. Our GiRPO still achieves the best balance between forgetting and task success rate.
We observed two sources of this gap.
(1) NPO-style methods struggle to control the forgetting strength. They tend to either diverge (e.g., exploding gradients) and hurt model utility, or forget too weakly[1-3]. But GiRPO’s reward penalty and advantage clipping on the pseudo rollout can handle this situation elegantly.
(2) A group-relative, task-calibrated signal in a single objective. The pseudo rollout is assigned a reward just below the worst real rollout (Eq. 4) and scored with the statistics of the real rollouts only (Eqs. 5-7). The forgetting signal is therefore calibrated to the policy’s current performance on each task, whereas NPO w/ GRPO apply a fixed, separately weighted loss (α) to all tasks. Isolating the statistics keeps the advantages of real rollouts identical to vanilla GRPO (Prop. 1), while naively including the forget trajectory in the group biases them (Thm. 1). Forgetting and utility are thus optimized in one objective, without splitting it into two losses.
[1] Maini, Pratyush, et al. "Tofu: A task of fictitious unlearning for llms." arXiv preprint arXiv:2401.06121 (2024).
[2] Shi, Weijia, et al. "Muse: Machine unlearning six-way evaluation for language models." International Conference on Learning Representations. Vol. 2025. 2025.
[3] Zhang, Ruiqi, et al. "Negative preference optimization: From catastrophic collapse to effective unlearning." arXiv preprint arXiv:2404.05868 (2024).