acceptodds
Under review as a conference paper at ICLR 2027

Trajectory Unlearning on LLM-based Agents

Abstract

Existing large language model (LLM) unlearning has focused primarily on removing specific knowledge, such as harmful facts, private data, or copyrighted content. However, as LLMs are increasingly deployed as autonomous agents, a fundamental yet overlooked problem emerges: beyond suppressing what an agent knows, an agent should not reproduce undesired behaviors through its action trajectories. In this work, we introduce trajectory-level unlearning, a new problem formulation that targets the removal of specific action trajectories in long-horizon agentic tasks, rather than factual knowledge. We identify two fundamental challenges that distinguish trajectory unlearning from knowledge unlearning: (1) our unlearning target is what the agent does, not what it says; and (2) trajectories are sequentially dependent action sequences that cannot be decomposed into isolated prompt-response pairs without losing inter-step structure. To address these challenges, we propose Group-injected Relative Policy Optimization (GiRPO), which injects forget trajectories into the policy rollout group with penalized rewards and isolates the normalization statistics, yielding a stable and bounded unlearning signal that does not corrupt gradient updates for normal task trajectories. We construct trajectory unlearning benchmarks from two application scenarios, household tasks (ALFWorld) and online shopping (WebShop), and design three complementary metrics for evaluating forgetting quality and model utility. Experiments on ALFWorld and WebShop demonstrate that GiRPO effectively unlearns target trajectories while preserving task success rates, outperforming existing knowledge-unlearning baselines on both forgetting quality and task utility.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Unlearning, Privacy, Safety

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bhLNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies trajectory unlearning: preventing an LLM agent from reproducing designated action sequences while preserving its ability to complete both target and other tasks. GiRPO augments policy rollout groups with penalized forget trajectories, isolates the reward-normalization statistics, and clips the negative forget advantages. Experiments on ALFWorld and WebShop show substantial reductions in trajectory repetition with comparatively well-preserved task success.

Critical strengths

GiRPO gives encouraging results on the stated sequence-avoidance task. On ALFWorld Clean, exact match falls from 33.9% to 1.4% while target success remains 88.4%, compared with 91.5% initially. On WebShop, target success improves while exact match falls.

Critical weaknesses
  1. The practical motivation for forgetting the selected trajectories remains unclear. I found the example in lines 76–91 lengthy and difficult to connect to the actual evaluation. It introduces a poisoned trajectory with a covert action, but does not clearly establish why the entire trajectory is the necessary unit of removal. More importantly, the ALFWorld and WebShop experiments do not sufficiently explain what makes the selected trajectories undesirable or why a user would want them forgotten. The appendix provides trajectory examples, but these do not clearly connect a concrete unwanted behavior to its removal after unlearning ( the judge prompt in Figure 11 measures overall task-solving similarity rather than whether a particular unwanted behavior survives. ). This leaves me uncertain whether the evaluation captures a practical behavior-removal problem or primarily measures avoidance of particular recorded action sequences.

  2. The baselines do not receive comparable opportunities to optimize the forgetting–success trade-off. GiRPO explicitly discourages forget trajectories while using online task rewards to learn successful alternatives on both target and untarget tasks. GA and NPO include retention losses, and DPO contrasts forget responses with substitute responses, but these objectives do not provide the same direct optimization of task success. GRPO, meanwhile, optimizes success only on untarget tasks without an explicit forgetting objective. NPO+GRPO is therefore the closest comparison, yet its GRPO component also excludes target tasks. The additional NPO v2 results partly address target-task coverage, but still do not match GiRPO’s online reward training. A comparison giving NPO+GRPO the same target and untarget tasks would help establish whether the reported advantage comes from GiRPO’s proposed group treatment rather than this difference in training opportunities.

Questions for authors

See weakness.

(1) What makes each designated trajectory undesirable, and which part must be removed? Can the existing rollout logs be evaluated for recurrence of that behavior or subtrajectory, including cases whose full sequence no longer matches? If the intended objective is strictly avoidance of recorded sequences, please clarify the scope of the safety claims accordingly.

(2) How does NPO+GRPO perform when its GRPO component receives the same target and untarget tasks, with a comparable rollout budget and the same exclusion of forget trajectories? This specific comparison would materially affect my assessment of GiRPO's contribution beyond access to alternative successful target-task behavior.

2 comments
AuthorsOpenReview ↗

Response to Reviewer bhLN

We sincerely thank the reviewer for acknowledging the benefits of GiRPO and for offering insightful suggestions. Our responses are provided below.

W1&Q1 The practical motivation for forgetting the selected trajectories remains unclear.

Why complete trajectories are the unit of an unlearning request. Each individual action in an undesirable trajectory is typically reasonable by itself (e.g., run a microwave), and the problem lies in the way actions are combined or ordered under a specific task. Removing the individual action or behavior would restrict normal agent behavior. We therefore choose complete trajectories as the unit of an unlearning request, because an undesirable execution path may consist entirely of individually legitimate actions whose combination or ordering is unwanted. This formulation allows the agent to suppress a designated execution path while preserving the general availability of its constituent actions.

What the selected trajectories are. In the ALFWorld and WebShop experiments, the forget trajectories are designated ones, as in unlearning benchmarks such as TOFU[1] and MUSE[2], where the forget set is designated rather than intrinsically harmful. They serve as a controlled testbed for whether a specific recorded trajectory can be removed while task utility is preserved. We do not claim these trajectories are intrinsically unsafe. To further illustrate our practical motivation, we have included an example of a poisoned trajectory in Figure 18 and its corresponding trajectory after unlearning in Figure 19.

Connecting a concrete behavior to its removal. To make this connection explicit, we added a poisoning experiment, where the undesirable behavior is putting an object into the garbagecan. Besides Exact Match, we measure the Behavior Recurrence Rate (BRR), the fraction of rollouts on the forget tasks that still exhibit the behavior. GiRPO reduces BRR from 92.5% to 62.9%, while NPO+GRPO barely changes it (88.5%) and NPO collapses. The LLM-as-Judge score differs from BRR by only a small margin (63.2% vs. 62.9%), so it also detects whether the behavior survives. The prompt and case studies are in Figs. 18-19.

Exact MatchAvg. Success RateBRRLLM as Judge
Base Model59.8%95.2%92.5%92.7%
NPO0%0%0%0%
NPO+GRPO26.9%97.3%88.5%88.6%
GiRPO4.9%94.4%62.9%63.2%

*For NPO, the unlearned model collapses and outputs “take take …” at every step, so BRR = 0 does not mean the model has forgotten the behavior.

W2&Q2 The baselines do not receive comparable opportunities to optimize the forgetting–success trade-off.

We added the requested comparison, NPO w/ GRPO. In these baselines, the GRPO component optimizes online task rewards on both target and untarget tasks, with the same rollout budget and the same forget-trajectory exclusion as GiRPO. Results are on ALFWorld-Clean (Qwen2.5-3B-Instruct), shown in the table below. Our GiRPO still achieves the best balance between forgetting and task success rate.

Clean TasksExact MatchSuccess Rate on Target TasksSuccess Rate on Untarget Tasks
Base Model33.9%91.5%87.3%
NPO w/ GRPO (target+untarget)12.5%85.8%86.8%
GiRPO1.4%88.4%88.5%

We observed two sources of this gap.

(1) NPO-style methods struggle to control the forgetting strength. They tend to either diverge (e.g., exploding gradients) and hurt model utility, or forget too weakly[1-3]. But GiRPO’s reward penalty and advantage clipping on the pseudo rollout can handle this situation elegantly.

(2) A group-relative, task-calibrated signal in a single objective. The pseudo rollout is assigned a reward just below the worst real rollout (Eq. 4) and scored with the statistics of the real rollouts only (Eqs. 5-7). The forgetting signal is therefore calibrated to the policy’s current performance on each task, whereas NPO w/ GRPO apply a fixed, separately weighted loss (α) to all tasks. Isolating the statistics keeps the advantages of real rollouts identical to vanilla GRPO (Prop. 1), while naively including the forget trajectory in the group biases them (Thm. 1). Forgetting and utility are thus optimized in one objective, without splitting it into two losses.

[1] Maini, Pratyush, et al. "Tofu: A task of fictitious unlearning for llms." arXiv preprint arXiv:2401.06121 (2024).

[2] Shi, Weijia, et al. "Muse: Machine unlearning six-way evaluation for language models." International Conference on Learning Representations. Vol. 2025. 2025.

[3] Zhang, Ruiqi, et al. "Negative preference optimization: From catastrophic collapse to effective unlearning." arXiv preprint arXiv:2404.05868 (2024).

AuthorsOpenReview ↗

Follow-up: Gentle Reminder Before the Discussion Deadline

Dear Reviewer bhLN,

With about 24 hours left in the discussion period, we would like to kindly follow up on our response.

For your concerns about the practical motivation (W1/Q1), we clarified why complete trajectories are the unit of an unlearning request, and added a poisoning experiment that connects a concrete behavior to its removal through a rule-based Behavior Recurrence Rate. For the baseline comparison (W2/Q2), we ran GA w/ GRPO (on both target and untarget tasks).

We would greatly appreciate your feedback on whether these additions address your concerns. Thank you very much for your valuable time and for helping us improve our work.

Reviewer_NMQbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors study trajectory unlearning in LLM-based agents, rather than knowledge unlearning as commonly considered in the literature. They inject designated forget trajectories into the rollout group as pseudo rollouts and assign them penalized rewards to discourage the policy from reproducing them. During advantage estimation, normalization statistics are computed using only real rollouts, while the negative advantages of pseudo rollouts are clipped to prevent excessively large updates. The policy is then updated using both real and pseudo rollouts. Experimental results demonstrate effective trajectory unlearning while somehow preserving task performance.

Critical strengths
  • The idea of trajectory unlearning in LLM-based agents is novel and interesting, with potential value in practical scenarios.

  • The experimental results show that the proposed method achieves some degree of trajectory unlearning on the evaluated tasks.

Critical weaknesses
  • The experimental results show that exact trajectory reproduction is successfully suppressed. However, since the LLM policy is probabilistic, it remains unclear whether similar behaviors are also successfully forgotten.

  • Negative reward signals could harm the overall performance of LLM-based agents, especially after repeated unlearning on the same agent. The experimental results also show some performance degradation.

  • The method itself is relatively simple, and introducing negative rewards may substantially reduce the performance of the original policy.

Questions for authors

Q1. Have you considered using a separate validation set? In the current setting, unlearning appears to be trained and evaluated using the same agent on the same tasks and environments, which may bias the results.

Q2. What happens if the trajectory designated for unlearning is similar to multiple behaviors learned by the agent? Does the method unlearn the underlying behavior or only suppress the exact trajectory?

Q3. The main components (reward penalties and isolated advantage estimation) do not appear to be limited to LLM-based agents. Why is the method evaluated only on LLM-based agents?

4 comments
AuthorsOpenReview ↗

Response to Reviewer NMQb - Part I

We sincerely thank the reviewer for recognizing the value of our trajectory unlearning idea and for providing constructive feedback. Our responses are provided below.

W1 The experimental results show that exact trajectory reproduction is successfully suppressed. However, since the LLM policy is probabilistic, it remains unclear whether similar behaviors are also successfully forgotten.

We sincerely thank the reviewer for the suggestions. The goal of trajectory unlearning is to forget the designated complete trajectory, rather than atomic actions or underlying capabilities. Each individual action or behavior the agent has learned is reasonable by itself, and it becomes problematic only under certain conditions, such as being combined or ordered in a poisoned way, or being inserted as irrelevant steps that reduce task efficiency. Suppressing the underlying actions in isolation would restrict the agent’s normal behavior, so they must be preserved. This is consistent with the preserved task success on untarget tasks (Tables 2-3).

Since the LLM policy is probabilistic, our Exact Match is computed over N trajectories sampled from the unlearned policy for each target task, and we use the base model’s Exact Match under the same sampling as a reference (natural variability, Sec. 4.1). The reported reductions are therefore measured against the model’s intrinsic randomness, rather than assuming deterministic behavior.

As an additional benefit, forgetting the complete trajectory also reduces the probability that the behavior appears under the specific condition (behavior in context). To directly examine a specific behavior, we conducted an additional experiment with Qwen2.5-3B-Instruct, where the target tasks are a subset of the Clean tasks. We take the behavior of putting an object into the garbagecan as an example; the prompt and a case study are shown in Figs. 18-19. The results show that, compared with NPO and NPO+GRPO, GiRPO substantially reduces the occurrence of the poisoned behavior by 29.5%. The Behavior Recurrence Rate (BRR) is the fraction of evaluation rollouts on the forget tasks in which the agent still exhibits the target behavior, computed as BRR = (# rollouts exhibiting the behavior) / (# rollouts). The LLM-as-Judge score also detects whether the behavior is present, and it differs from BRR by only a small margin, which shows that the LLM-as-Judge metric is also effective.

Exact MatchAvg. Success RateBRRLLM as Judge
Base Model59.8%95.2%92.5%92.7%
NPO0%0%0%0%
NPO+GRPO26.9%97.3%88.5%88.6%
GiRPO4.9%94.4%62.9%63.2%

*For NPO, the unlearned model collapses and outputs “take take …” at every step, so BRR = 0 does not mean the model has forgotten the behavior.

AuthorsOpenReview ↗

Response to Reviewer NMQb - Part II

W2 Negative reward signals could harm the overall performance of LLM-based agents, especially after repeated unlearning on the same agent. The experimental results also show some performance degradation.

We thank the reviewer for the feedback. It is worth noting that unlearning inherently involves a trade-off between forget quality and task performance. In our experiments, GiRPO achieves a more favorable balance than other baselines. On ALFWorld-Clean (Table 2), GA and NPO reduce target success to only 0.9% (-90.6%) and 1.2% (-90.3%), and GA also lowers untarget success to 76.6% (-10.7%). GiRPO keeps target success at 88.4% and untarget success at 88.5% (+1.2%).

The negative reward signal is applied only to the forget trajectories (Eq. 4). The normalization statistics are computed over real rollouts only (Eqs. 5–6), so real-rollout advantages are identical to vanilla GRPO (Prop. 1). The forget advantage is lower-bounded by −λ (Eq. 7), which bounds the magnitude of the negative forget advantage, preventing near-zero reward variance from producing arbitrarily large advantage coefficients and helping stabilize optimization (Prop. 4, Appendix A.4).

We also ran two additional experiments.

  1. Continual Unlearning

We sequentially unlearned Clean, Heat, and Cool. Target success stays at or above 88.7% throughout. After the last round, target success is +1.8% relative to the base model, and untarget success is only 0.6% lower.

ModelsTarget TasksTarget Success RateExact MatchUntarget Success Rate
BeforeClean91.5%33.9%87.3%
GiRPOClean88.4%(-3.1%)1.4%88.5%(+1.2%)
BeforeClean+Heat91.6%30.8%86.7%
GiRPOClean+Heat88.7%(-2.9%)2.4%85.1%(-1.6%)
BeforeClean+Heat+Cool89.5%27.2%87.1%
GiRPOClean+Heat+Cool91.3%(+1.8%)3.0%86.5%(-0.6%)
  1. Ablation Study on Reward Penalty δ\delta

Our default reward penalty is a small value (δ=1\delta=1) and does not noticeably hurt overall performance. In our ablation study, varying the reward penalty (without the clipping threshold λ) within 10 has only a minor impact on the success rate, which stays above 88%. Only when the penalty is increased to 100 does the success rate drop sharply, to 72.12%.

δ\deltasuccess rateexact match
188.41%1.42%
589.32%2.04%
1088.76%1.53%
10072.12%0.00%

W3 The method itself is relatively simple, and introducing negative rewards may substantially reduce the performance of the original policy.

It is worth noting that unlearning generally involves a trade-off between forget quality and task performance. GiRPO achieves a more favorable balance than baselines. Please see our response to W2 for a detailed explanation.

GiRPO is simple, yet effective and carefully designed: the reward penalty and the isolated advantage estimation are each motivated by a specific failure mode (Prop. 1, Theorem 1), and a more complicated algorithm is not necessarily better. Suppressing a forget trajectory inevitably requires a negative signal, as in conventional LLM unlearning methods (e.g., GA, NPO). The goal of our work is therefore not to avoid negative signals, but to make them better controlled, so as to reduce utility degradation and achieve a better trade-off.

Q1. Have you considered using a separate validation set?

(1) Cross-environment evaluation (ALFWorld to WebShop)

ALFWorld and WebShop are closed-domain environments, and our base models have been post-trained specifically for the tasks in their respective environments, so even the base model performs poorly outside them. Following the reviewer’s suggestion, we conducted a cross-environment evaluation, testing the ALFWorld agent before and after unlearning on WebShop.

Scoresuccess rate
Before0.0591.5%
NPO0.0420.5%
GiRPO0.0631.3%

The base model itself performs poorly in this out-of-distribution environment, and unlearning with GiRPO does not further reduce its performance too much.

(2) Held-out evaluation on ALFWorld (valid_seen / valid_unseen)

We also evaluated on the held-out ALFWorld splits. The rooms in valid_seen appear in the training set, while those in valid_unseen do not.

valid_seenvalid_unseen
Before unlearning93.8%94.8%
NPO73.6%69.7%
GiRPO89.8%90.8%

It shows GiRPO outperforms NPO on maintaining performance on held-out ALFWorld datasets.

(3) General capability evaluation (GSM8K)

Finally, we evaluated the general capability of the model before and after unlearning on GSM8K.

Accuracy
Before unlearning86.13%
NPO85.44%
GiRPO85.90%

Overall, the results show that GiRPO does not harm the model’s general capability that much.

AuthorsOpenReview ↗

Response to Reviewer NMQb - Part III

Q2 What happens if the trajectory designated for unlearning is similar to multiple behaviors learned by the agent? Does the method unlearn the underlying behavior or only suppress the exact trajectory?

We sincerely thank the reviewer for the suggestions. The goal of trajectory unlearning is to forget the designated complete trajectory, rather than atomic actions or underlying capabilities. Each individual action or behavior the agent has learned is reasonable by itself, and it becomes problematic only under certain conditions, such as being combined or ordered in a poisoned way, or being inserted as irrelevant steps that reduce task efficiency. Suppressing the underlying actions in isolation would restrict the agent’s normal behavior, so they must be preserved. This is consistent with the preserved task success on untarget tasks (Tables 2-3).

As an additional benefit, forgetting the complete trajectory also reduces the probability that the behavior appears under the specific condition (behavior in context). To directly examine a specific behavior, we conducted an additional experiment with Qwen2.5-3B-Instruct, where the target tasks are a subset of the Clean tasks. We take the behavior of putting an object into the garbagecan as an example; the prompt and a case study are shown in Figs. 18-19. The results show that, compared with NPO and NPO+GRPO, GiRPO substantially reduces the occurrence of the poisoned behavior by 29.5%. The Behavior Recurrence Rate (BRR) is the fraction of evaluation rollouts on the forget tasks in which the agent still exhibits the target behavior, computed as BRR = (# rollouts exhibiting the behavior) / (# rollouts). The LLM-as-Judge score also detects whether the behavior is present, and it differs from BRR by only a small margin, which shows that the LLM-as-Judge metric is also effective.

Exact MatchAvg. Success RateBRRLLM as Judge
Base Model59.8%95.2%92.5%92.7%
NPO0%0%0%0%
NPO+GRPO26.9%97.3%88.5%88.6%
GiRPO4.9%94.4%62.9%63.2%

*For NPO, the unlearned model collapses and outputs “take take …” at every step, so BRR = 0 does not mean the model has forgotten the behavior.

Q3 The main components (reward penalties and isolated advantage estimation) do not appear to be limited to LLM-based agents. Why is the method evaluated only on LLM-based agents?

We agree that reward-relative penalization and isolated normalization are not specific to LLMs. We restrict our evaluation to LLM-based agents for three reasons.

(i) The threat we study, i.e., poisoned or deprecated trajectories embedded during SFT/RL post-training and triggered in other tasks, arises in LLM-agent training pipelines.

(ii) Our metrics (Exact Match, LLM-as-Judge) require discrete symbolic action sequences. In continuous control, any specific trajectory has zero probability, so “reproducing a forget trajectory” must first be redefined, which we view as a separate problem.

(iii) The relevant baselines (GA, DPO, NPO) are LLM unlearning methods, which gives a meaningful comparison in this setting.

Thus, extending trajectory unlearning to conventional RL agents is therefore non-trivial: it requires a new definition of the forgetting target, new evaluation metrics, and further adaptation of the method, which we consider a separate line of work and leave to future research. We have added a sentence in the Conclusion stating that extending trajectory unlearning to non-LLM policies is future work.

AuthorsOpenReview ↗

Follow-up: Gentle Reminder Before the Discussion Deadline

Dear Reviewer NMQb,

With about 24 hours left in the discussion period, we would like to kindly follow up on our response.

For your concerns about similar behaviors (W1/Q2), we added a poisoning experiment with a Behavior Recurrence Rate (BRR) that goes beyond exact trajectory match. For the potential harm of negative rewards and repeated unlearning (W2/W3/Q1), we added the continual unlearning, the reward-penalty ablation, held-out evaluation (valid_seen/valid_unseen), cross-environment evaluation, and GSM8K experiments.

We would greatly appreciate your feedback on whether these additions address your concerns. Thank you very much for your valuable time and for helping us improve our work.

Reviewer_gStVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies trajectory unlearning for LLM agents, aiming to suppress designated action sequences while preserving task completion. The proposed GiRPO method extends GRPO by injecting forget trajectories as negatively rewarded pseudo-rollouts, computing normalization statistics separately, and clipping negative advantages.

Critical strengths
  1. The problem is practically relevant. Suppressing an undesirable action sequence while preserving legitimate uses of its constituent actions is a useful objective for agent systems

  2. The reported results show meaningful improvements in the forgetting–utility trade-off. On ALFWorld Clean, exact match drops from 33.9% to 1.4%, while task success changes from 91.5% to 88.4%. On WebShop, target-task success improves from 68.1% to 76.2% alongside reduced replay (Tables 2–3).

Critical weaknesses
  1. The evaluation does not directly establish removal of the undesirable behavior. Exact match measures whether an entire recorded sequence is reproduced, so changing an irrelevant step can improve this metric while leaving the problematic behavior intact. LLM similarity provides a broader comparison, but it does not directly measure whether the undesirable action pattern still occurs in its triggering context (Sec. 3.1). These metrics support the narrow objective of reducing trajectory replay, but the stronger claims about removing context-dependent unsafe behavior need more direct evidence (Secs. 1–2).

  2. The explanation of GiRPO’s advantage over existing methods is not convincing. Sec. 3.2 argues that conventional unlearning loses inter-step dependencies. However, the baseline prompts already include preceding actions (Sec. 4.1), and GiRPO’s objective also sums per-action terms (Eq. 9). The results show an empirical advantage, but do not establish that preserving long-horizon dependencies explains it. This leaves the claimed methodological distinction insufficiently supported.

Questions for authors
  1. Validation of the main claim: Can you measure how often the designated undesirable action pattern still occurs when surrounding steps change, including on other tasks with similar intermediate states? This would clarify whether reduced replay corresponds to the behavioral removal motivating the paper.

  2. Clarification of the method comparison: What dependency information do the history-conditioned baselines lose? Do the alternative implementations in Appendix A.3 include trajectory-level gradient ascent combined with the same GRPO utility objective and comparable clipping and weighting? Such a comparison would help identify what accounts for GiRPO’s gains.

3 comments
AuthorsOpenReview ↗

Response to Reviewer gStV - Part I

Thank you for recognizing the significance of the trajectory unlearning problem we study. We sincerely appreciate the reviewer's insightful feedback and constructive suggestions. Our responses are provided below.

W1 & Q1 The evaluation does not directly establish removal of the undesirable behavior.

The goal of trajectory unlearning is to forget the designated complete trajectory, rather than atomic actions or underlying capabilities. Each individual action or behavior the agent has learned is reasonable by itself, and it becomes problematic only under certain conditions, such as being combined or ordered in a poisoned way, or being inserted as irrelevant steps that reduce task efficiency. Suppressing the underlying actions in isolation would restrict the agent’s normal behavior, so they must be preserved. This is consistent with the preserved task success on untarget tasks (Tables 2-3).

As an additional benefit, forgetting the complete trajectory also reduces the probability that the behavior appears under the specific condition (behavior in context). To directly examine a specific behavior, we conducted an additional experiment with Qwen2.5-3B-Instruct, where the target tasks are a subset of the Clean tasks. We take the behavior of putting an object into the garbagecan as an example; the prompt and a case study are shown in Figs. 18-19. The results show that, compared with NPO and NPO+GRPO, GiRPO substantially reduces the occurrence of the poisoned behavior by 29.5%. The Behavior Recurrence Rate (BRR) is the fraction of evaluation rollouts on the forget tasks in which the agent still exhibits the target behavior, computed as BRR = (# rollouts exhibiting the behavior) / (# rollouts). BRR is determined by a rule-based matcher. ****The LLM-as-Judge score also detects whether the behavior is present, and it differs from BRR by only a small margin, which shows that the LLM-as-Judge metric is also effective.

Exact MatchAvg. Success RateBRRLLM as Judge
Base Model59.8%95.2%92.5%92.7%
NPO0%0%0%0%
NPO+GRPO26.9%97.3%88.5%88.6%
GiRPO4.9%94.4%62.9%63.2%

*For NPO, the unlearned model collapses and outputs “take take …” at every step, so BRR = 0 does not mean the model has forgotten the behavior.

W2 The explanation of GiRPO’s advantage over existing methods is not convincing.

  1. What challenge ➋ is (and is not) claiming. Challenge ➋ (Long-Horizon Dependencies) characterizes our proposed trajectory unlearning task, not a deficiency of specific baselines. In trajectory unlearning, the forgetting target is a sequence whose steps depend on the interaction history. It is a statement about how trajectory unlearning differs from knowledge unlearning, and we do not claim that it alone explains GiRPO’s gains. We have revised the sentence in Sec. 3.2 that contrasted GiRPO with methods that “treat a trajectory as text-based prompt-response pairs”, to prevent potential misinterpretation.
  2. We deliberately make the baseline prompts include all preceding actions (Sec. 4.1, Fig. 16) for fair comparison. We deliberately adapted the knowledge-unlearning baselines to the trajectory unlearning setting, conditioning every step on its full history, so that they are compared fairly rather than as strawmen. Even so, they show a clear trade-off between forgetting and utility (Table 2, Figs. 4–5).
  3. Our point is about the form of the learning signal. Even with history-conditioned inputs, GA/DPO/NPO still optimize a text-likelihood objective on fixed recorded sequences, with no notion of task outcome. GiRPO instead assigns a single outcome-level (trajectory-level) reward to the whole forget trajectory, judged relative to the policy’s own rollouts on the same task, and optimizes it jointly with task success. We have revised Sec. 3.2 accordingly, removing statements that suggest the baselines lose inter-step dependencies.
AuthorsOpenReview ↗

Response to Reviewer gStV - Part II

Q2 Clarification of the method comparison: What dependency information do the history-conditioned baselines lose? Do the alternative implementations in Appendix A.3 include trajectory-level gradient ascent combined with the same GRPO utility objective and comparable clipping and weighting? Such a comparison would help identify what accounts for GiRPO’s gains.

Q2 (a): For the first sub-question, please refer to W2 for our detailed response.

Q2 (b): Trajectory-level GA + the same GRPO objective. We have now added this comparison. Results on ALFWorld-Clean (Qwen2.5-3B-Instruct). GiRPO achieves the best balancing between forgetting and task success rate. GA w/ GRPO is even less stable across task types: its success drops to 64.6% on Look and 83.7% on Heat.

Clean TasksExact MatchSuccess Rate on Target TasksSuccess Rate on Untarget Tasks
Base Model33.9%91.5%87.3%
GA w/ GRPO4.7%88.5%84.4%
GiRPO1.4%88.4%88.5%

We observed three sources of this gap.

(1) Conflicting signals. A forget trajectory can itself complete the task successfully, so when it is re-sampled as a real rollout during GRPO, the utility term still rewards it on GA w/ GRPO baselines. In GiRPO, the forget trajectory is injected into the rollout group as a pseudo rollout and scored against the real rollouts. Real rollouts identical to a forget trajectory are discarded (Eq. 3), so the utility reward and the forgetting penalty no longer act on the same trajectory.

(2) GA-style methods struggle to control the forgetting strength. They tend to either diverge (e.g., exploding gradients) and hurt model utility, or forget too weakly[1-3]. But GiRPO’s reward penalty and advantage clipping on the pseudo rollout can handle this situation elegantly.

(3) A group-relative, task-calibrated signal in a single objective. The pseudo rollout is assigned a reward just below the worst real rollout (Eq. 4) and scored with the statistics of the real rollouts only (Eqs. 5-7). The forgetting signal is therefore calibrated to the policy’s current performance on each task, whereas GA w/ GRPO apply a fixed, separately weighted loss (α) to all tasks. Isolating the statistics keeps the advantages of real rollouts identical to vanilla GRPO (Prop. 1), while naively including the forget trajectory in the group biases them (Thm. 1). Forgetting and utility are thus optimized in one objective, without splitting it into two losses.

[1] Maini, Pratyush, et al. "Tofu: A task of fictitious unlearning for llms." arXiv preprint arXiv:2401.06121 (2024).

[2] Shi, Weijia, et al. "Muse: Machine unlearning six-way evaluation for language models." International Conference on Learning Representations. Vol. 2025. 2025.

[3] Zhang, Ruiqi, et al. "Negative preference optimization: From catastrophic collapse to effective unlearning." arXiv preprint arXiv:2404.05868 (2024).

AuthorsOpenReview ↗

Follow-up: Gentle Reminder Before the Discussion Deadline

Dear Reviewer gStV,

With about 24 hours left in the discussion period, we would like to kindly follow up on our response.

Regarding your concern that the metrics do not directly establish removal of the undesirable behavior (W1/Q1), we ran a poisoning experiment with a rule-based Behavior Recurrence Rate (BRR), where GiRPO reduces BRR by 29.5%. Regarding the fairer baseline (W2/Q2), we added NPO w/ GRPO baseline (on both target and untarget tasks).

We would greatly appreciate your feedback on whether these additions address your concerns. Thank you very much for your valuable time and for helping us improve our work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comments: New Evidence and Clarifications

We sincerely thank all the reviewers for their constructive feedbacks! We address individual concerns and questions via replies to the respective reviews below. Here we would like to highlight a few key points:

  1. Behavior-level evidence. Our new poisoning experiment shows that unlearning the designated trajectory also reduces the undesirable behavior in context, with GiRPO achieving a 29.5% reduction in the Behavior Recurrence Rate.

  2. Fairer baselines. We added NPO w/ GRPO baselines, where GRPO optimizes online task rewards on both target and untarget tasks. In our experiments, GiRPO achieves the best balance between forgetting and utility. And our GiRPO won't suffer from the over-forgetting and model collapse in GA/NPO alone.

  3. Our contribution. We would like to highlight the trajectory unlearning problem itself, which shifts the unlearning target from what a model says to what an agent does. We believe this is an important and timely level of granularity for agentic unlearning.

We hope this clarification helps position and clarify the contribution of our work and therefore hope the reviewers can judge the work in that light.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.