acceptodds
Under review as a conference paper at ICLR 2027

Remembering How to Act through Greedy Action Consolidation in Continual Reinforcement Learning

Abstract

For value-based continual control, retaining a learned decision need not entail preserving the action values that originally supported it, since greedy action selection depends on their relative ordering rather than their absolute magnitudes. We propose Greedy Action Consolidation, or GAC, which targets the gaps that support historical greedy actions to mitigate catastrophic forgetting while allowing action value estimates to evolve. At the core of GAC lies the continual action gap, which measures the action value difference, under the updated function, between a historical greedy action and its best competing action, and is regularized to remain at least a prescribed fraction of it's historical action gap. Theoretically, we derive an upper bound on return discrepancy in terms of the GAC regularizer and the distribution of historical action gaps, and show that GAC strictly enlarges the pointwise admissible action value set relative to exact value preservation as instantiated by Value Cloning, affording greater freedom for value adaptation. Experiments on MinAtar and Forager demonstrate the effectiveness of GAC across semi-continual and continual reinforcement learning settings, suggesting that the action gaps provide a parsimonious substrate for remembering how to act in continual reinforcement learning.

Primary area:
reinforcement learning
Keywords:
continual reinforcement learning, action gap, catastrophic forgetting

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_YbtJOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

To address catastrophic forgetting in CRL and the overly restrictive nature of conventional value cloning, this manuscript proposes GAC from the perspective of decision preservation rather than value preservation. By defining the continual action gap and constructing a hinge regularization loss based on historical action gaps, the manuscript relaxes constraints on the absolute magnitudes of Q values while maintaining the relative advantage of historical greedy actions. The manuscript provides relevant theoretical analysis and validates the effectiveness of the method on the MinAtar and Forager environments.

Critical strengths
  1. The manuscript introduces a reasonable and practically meaningful research entry point. It explicitly distinguishes numerical prediction stability from behavioral stability, offering intuitive observations that provide compelling motivation for the proposed method.
  2. The method design is concise and intuitive. It does not require modifying the core objective of base TD learning, exhibiting good modular potential. The experiments also adequately validate its integration capability with DQN and PT-DQN.
  3. The theoretical analysis demonstrates good completeness, providing a formal explanation for the design motivation with an overall clear derivation process.
  4. The experimental scenarios cover both settings where task boundaries are known and settings where task boundaries are unobserved. Valuable ablation studies are conducted around the retention coefficient, memory capacity, and action alignment mechanism.
Critical weaknesses
  1. The manuscript lacks direct policy preservation and action ranking baselines, which in my view leaves the core contribution insufficiently clear. Although the manuscript compares against Value Cloning with the policy cloning term removed, this only represents a numerical value retention baseline and cannot represent methods that directly preserve historical policies.
  2. In Forager, partial observability causes perceptual aliasing and periodic goal conflicts, where the same observation may correspond to opposite optimal actions across different phases. In my assessment, the experimental design and evaluation protocol in the manuscript do not sufficiently demonstrate that the method genuinely mitigates catastrophic forgetting on historical tasks.
  3. The theoretical results rely on relatively strong conditions, and their guarantees for practical continual learning performance remain rather limited in my view.
  4. In RL, historical greedy actions are not necessarily optimal actions. Subsequent training that corrects earlier erroneous action orderings should ideally serve as positive knowledge updating, yet under the Behavioral Forgetting metric defined in the manuscript, this is counted as behavioral forgetting. Furthermore, due to Q value overestimation or training instability, erroneous actions may also produce large historical action gaps. Applying stronger consolidation to them could inadvertently entrench suboptimal policies and induce negative transfer. Therefore, I consider both the Behavioral Forgetting metric and the historical gap weighting mechanism to be fragile.
  5. The empirical gains in the current manuscript are inconsistent, and the verification of statistical reliability and generalization remains insufficient. For example, on Space Invaders in MinAtar, DQN+GAC underperforms Value Cloning and exhibits noticeably larger variance, while on Forager, DQN+GAC also lags behind Online EWC. In addition, I find that the random seeds, experimental scale, and statistical testing are relatively weak.
  6. The manuscript does not clearly explain the memory replacement strategy or how conflicting historical constraints for the same state are handled. Furthermore, I note that the manuscript lacks fair comparisons under fixed memory byte budgets and computational time budgets.
Questions for authors
  1. Regarding my first weakness, could the authors include direct behavioral preservation baselines to clarify the actual empirical gain brought by the specific numerical values of historical action gaps?
  2. Regarding my second weakness, in Forager or environments with conflicting goals, how exactly does GAC handle contradictory action gap constraints produced for the same observation across different phases? Could the authors provide controlled historical task reevaluation or recovery speed metrics?
  3. Regarding my third and fourth weaknesses, could the authors report the empirical distributions of historical action gaps and actual GAC loss values during real training to illustrate whether Theorem 1 is tight on empirical data?
  4. Regarding my fifth and sixth weaknesses, could the authors clarify the specific replacement strategy for the auxiliary memory and provide the actual memory footprint alongside training runtime overhead?
4 comments

We thank the reviewer for the thoughtful and technically substantive feedback. We address the concerns point by point below.

Weakness 1 & Question 1. could the authors include direct behavioral preservation baselines to clarify the actual empirical gain brought by the specific numerical values of historical action gaps?

We would like to clarify that our evaluation already includes FAME [1], which provides a principled policy-level preservation baseline for value-based continual reinforcement learning.

Policy Preservation through FAME. FAME transforms each task-specific Q-function into a Boltzmann policy,

\frac{\exp(Q_i(s,a)/\tau)} {\sum_{a'}\exp(Q_i(s,a')/\tau)},$$ and consolidates historical policies by minimizing their cumulative KL divergence from a shared meta policy. Its value-based formulation can be expressed as $$Q_k^M = \arg\min_{\tilde Q_k^M} \sum_{i=1}^{k}\sum_s \mu_i^{Q_i}(s) \left[ \mathrm{KL} \left( \pi^{Q_i}(\cdot\mid s) \Vert \tilde{\pi}_k^M(\cdot\mid s) \right) \right].$$ Here, $\mu_i^{Q_i}(s)$ weights states according to the distribution associated with the historical task-specific Q-function, while $\tilde{\pi}_k^M$ denotes the Boltzmann policy induced by the candidate meta Q-function $\tilde Q_k^M$. Thus, FAME explicitly consolidates historical action distributions through policy distillation, rather than directly preserving numerical action gaps or value magnitudes as Value Cloning does. Our existing MinAtar comparison already includes both FAME components, since its meta learner and fast learner respectively serve knowledge integration and task adaptation. As reported in Table 1, DQN with GAC achieves higher normalized lifetime AUC than both FAME components across all three environments. |Method|Breakout|Space Invaders|Freeway| |-|-:|-:|-:| |DQN w/ GAC|**14.858 ± 0.365**|**88.282 ± 19.797**| **26.121 ± 2.988** | |DQN w/ FAME (meta learner)|10.170 ± 0.271|16.798 ± 4.114|11.542 ± 3.351 | |DQN w/ FAME (fast learner)|4.737 ± 0.770|26.338 ± 1.002|11.077 ± 0.095| These results provide an empirical comparison against explicit policy-level consolidation in value-based continual reinforcement learning and demonstrate the effectiveness of directly preserving historical greedy action gaps. **Contribution of Historical Action Gap Magnitudes.** Our retention-coefficient ablation in Figure 4a further examines the contribution of the specific numerical values of historical action gaps. In particular, $\rho=0$ only penalizes negative continual action gaps, whereas $\rho=0.5$ additionally enforces the retention of at least half of each historical action gap. The consistently higher normalized AUC achieved with $\rho=0.5$ across all three MinAtar environments provides complementary evidence that preserving the specific numerical values of historical action gaps offers additional benefits. > **Weakness 2 & Question 2**. how exactly does GAC handle contradictory action gap constraints produced for the same observation across different phases? Could the authors provide controlled historical task reevaluation or recovery speed metrics? We thank the reviewer for highlighting this important issue. We clarify that GAC does not resolve perceptual aliasing or explicitly reconcile contradictory historical constraints. Specifically, when the same observation corresponds to different historical greedy actions $a$ and $b$, GAC may impose mutually incompatible constraints, $$Q(s,a)-Q(s,b)\geq \rho M_1(s)>0, \qquad Q(s,b)-Q(s,a)\geq \rho M_2(s)>0.$$ Nevertheless, GAC offers greater flexibility than Value Cloning. As established by Theorem 2, its feasible set strictly contains that of exact value preservation. This additional flexibility allows the $Q$-function to adapt more freely to changing reward mappings, although it does not guarantee the resolution of intrinsically conflicting decisions. Importantly, our Forager experiments are not intended to demonstrate that GAC solves state aliasing, but rather to empirically examine the practical advantage of the larger feasible set established by Theorem 2 under partial observability. **Controlled Historical-Task Reevaluation.** To examine this issue empirically, we additionally evaluate historical-task performance in the two-biome Forager environment. While training under reward mapping 1, we periodically evaluate the current agent under the conflicting reward mapping 2 and compute its normalized AUC, where higher values indicate better historical-task retention. |Method|Norm. AUC| |-|-:| |DQN w/ GAC|**−0.2707± 0.0431**| |DQN w/ Online EWC| −0.3780±0.0710| Both methods exhibit negative normalized AUC, confirming that conflicting reward mappings induce substantial interference. Nevertheless, GAC achieves better historical-task retention than Online EWC. **References** [1] Sun et al. Principled Fast and Meta Knowledge Learners for Continual Reinforcement Learning. ICLR, 2026.

Weakness 3,4 & Question 3. relatively strong conditions...historical greedy actions are not necessarily optimal actions...report empirical historical action gap distributions and actual GAC losses to assess whether Theorem 1 is tight on empirical data?

Theoretical Assumptions. Theorem 1 relies on a reference distribution matching the historical policy's discounted state occupancy to establish the return discrepancy bound. This is a theoretical assumption, not a requirement of practical GAC optimization. In model-free online continual RL, exact historical occupancy is generally unavailable, though approximable using retained data. GAC therefore uses auxiliary memory states sampled from the replay buffer without requiring them to follow this occupancy. We do not claim an unconditional return guarantee for arbitrary replay distributions.

Historical Action Reliability. Section 3 clarifies that a^⋆\hat a^\star denotes a learned greedy action, not necessarily an optimal one. Since optimal action labels are generally unavailable in model-free RL, GAC uses the learned greedy policy as a reference for behavioral retention. Accordingly, behavioral forgetting measures deviations from historical learned behavior, not necessarily harmful performance degradation, and may count beneficial corrections. Greedy action alignment requires agreement between online and lagged target networks and retains the smaller of their action gaps. This filtering reduces sensitivity to transient action-ranking instability and potentially overestimated gaps, although cannot guarantee optimality.

Empirical Validation. We add empirical analyses of Theorem 1. First, historical action gap distributions across Breakout, Space Invaders, and Freeway reveal task-dependent variation. Second, using actual auxiliary memory states, we track behavioral forgetting and its empirical upper bound throughout training. This reflects the practical regularizer's state distribution rather than assuming theoretical historical occupancy. The bound remains around 0.10.1–0.50.5, while observed behavioral forgetting stays within approximately 0.0020.002–0.030.03. Though conservative rather than numerically tight, it provides a non-vacuous characterization under the practical distribution.

We also report actual GAC losses from auxiliary minibatches used in MinAtar gradient updates, characterizing the magnitude and dispersion of violations.

Logged UpdatesMeanStd.MedianP95Max
6,7239.4301e-41.0420e-36.9456e-42.8211e-31.1462e-2

Revision. Added to Section 5.3.

Weakness 5. The empirical gains in the current manuscript are inconsistent...the verification of statistical reliability and generalization remains insufficient.

Additional Generalization Evidence. See our response to Reviewer ZzXr, Weakness 2, for additional MinAtar sequences and the challenging unending four-biome Forager evaluation. GAC achieves an average normalized AUC of 0.0970.097 over 15 independent runs in this environment, compared with 0.0890.089 for Online EWC.

Furthermore, we extend GAC to hierarchical value-based learning through a Deep Option Network (DON) based on the options framework and intra-option value learning of [2]. GAC is applied to both the option-value function QΩ(s,o)Q_{\Omega}(s,o) and intra-option action-value functions Qo(s,a)Q_o(s,a).

MethodNorm. AUC
DON0.022±0.110
DON w/ GAC0.225±0.381

These preliminary results suggest that GAC can benefit hierarchical value learning by consolidating decisions across multiple levels of temporal abstraction.

Revision. We have added the unending four-biome Forager results to Section 5.2, additional MinAtar evaluations to Appendix D, and DON experiments to Appendix F.

Weakness 6. ...the memory replacement strategy or how conflicting historical constraints for the same state are handled ... lacks fair comparisons under fixed memory byte budgets and computational time budgets.

Auxiliary Memory Management. For memory construction and replacement procedures, see our response to Reviewer ZzXr, Weakness 1, and Appendix B pseudocode. Both methods use lazy reservoir sampling over replay-buffer states, while GAC additionally applies greedy action alignment.

Memory Footprint and Runtime. See our response to Reviewer H1Pb, Weaknesses 3b–3c, for per-record storage costs and clarification that memory scaling matches record counts rather than byte budgets. We also report actual Forager memory footprints.

MethodMemory Footprint
GAC98.05 KiB
Online EWC161.03 KiB

For wall-clock profiling, see our response to Reviewer H1Pb, Weakness 3a, for MinAtar measurements under matched environment-step budgets. Forager exhibits the same relative wall-clock ordering.

References

[2] Sutton et al. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 1999.

Thanks for your thorough and candid response. The new results are helpful. Your clarifications also resolve my questions about the auxiliary memory mechanism and experimental fairness. Some conceptual limitations still remain, such as conflicting constraints under partial observability and the risk of locking in suboptimal historical actions. Even so, I appreciate your honest discussion of these boundaries and your solid empirical work. I will raise my rating.

Thank you for taking the time to review our rebuttal and reconsider your assessment of our work. We sincerely appreciate your thoughtful feedback and careful consideration.

Reviewer_H1PbOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper presents a method to address cathastrophic forgetting in task agnostic continual reinforcement learning, by assessing whether learning a new task changes the expected return of actions applied to a state revisited (previously seen when learning a previous task) from the expected return of the best competing action obtained with current Q-function. Instead of forcing to preserve exact action values or state-action transitions, the method relies on a so called "continual action gap," which measures the margin between the value of a historical greedy action and its best competing action under the updated Q-function. Intuitively, the method posits that an agent has not forgotten how to act in a state as long as the historical greedy action remains the optimal choice (i.e., the action gap remains positive), regardless of whether the absolute numerical values of the Q-function have drifted. That is, the current Q-function still evaluates the historical greedy action with the better expected return. The method posits an elegant approach in decoupling: by regularizing the continual action gap rather than enforcing exact value cloning. With this approach, the state-action transitions may change, and the Q-function is flexible in adapting to new tasks, as long as the gap remains positive.

The method is validated through experiments in two environments (MinAtar and Forager), covering both semi-continual and fully continual settings. The evaluation reuses established protocols and adds new metrics—specifically "behavioral forgetting" and "lifetime return"—to demonstrate that GAC effectively mitigates decision-level forgetting without sacrificing plasticity.

Critical strengths

1- The theoretical formulation of the Continual Action Gap (Eq. 2) and the resulting GAC regularizer is elegant and mathematically sound. 2- It successfully decouples decision retention from exact value preservation 3- It captures the concept of decision stability being independent of value magnitude fixing the historical action a^j⋆\hat{a}^\star_j and evaluating its margin under the new Q-function QθkQ_{\theta_k}. 4- The assumption of a "fixed deterministic tie-breaking rule" is a rigorous formal choice, without which a gap of exactly zero would make the behavioral forgetting indicator function \mathbb{I}{\hat{a}^\star_k(s) \neq \hat{a}^\star_j(s)} discontinuous. 5- The proof of Theorem 2 elegantly demonstrates that the GAC feasible set strictly contains the exact Value Cloning set by constructing a shifted vector $Q_{(s,a)}+C.

​

Critical weaknesses

The paper requires some work to bring more clarity: In continual learning, there is an overlap in state spaces for different tasks. This is what actually leads to forward transfer and/or catastrophic forgetting. 1- Clarification on the Reference Distribution μj\mu_j (Section 4.1): Lines 156-158 define the reference state distribution μj\mu_j as being "fixed with θj\theta_j and independent of θk\theta_k ." This phrasing can be easily misinterpreted. It is clear from the context (and the subsequent instantiation via an empirical replay buffer) that μj\mu_j is meant to be a fixed reference measure (a constant with respect to the optimization variables θk\theta_k) used to evaluate behavioral forgetting. However, the word "independent" might incorrectly lead readers to believe there is an assumption that the environment's state transitions do not overlap across tasks, or that the agent never visits states in μj\mu_j​ while operating under θk\theta_k.

Recommendation: The authors should rephrase this to explicitly state that μj\mu_j is "treated as a fixed constant during the optimization of θk\theta_k​" or "does not depend on the current parameters of θk\theta_k ." Clarifying this will prevent confusion regarding the assumption of shared state spaces across tasks.

2- Clarification on the Continual Action Gap Formulation (Eq. 2): The definition of the continual action gap (Eq. 2) evaluates the margin between the historical greedy action and the best competing action under the current Q-function. While mathematically sound, the text should clarify why the gap is measured against the "best competing action" rather than simply checking if the historical action matches the "current greedy action." Explicitly stating that this formulation measures the decision margin (or the distance to the decision boundary) and connects to the classical action gap literature would help readers understand why this relative margin is a more robust target for the hinge regularizer than simple action matching.

The practical implementation and computational margin versus value cloning. 3- The proposed method requires computing the historical greedy action over the state distribution μj\mu_j, which is sampled from the replay buffer. At the completion of task jj, the network parameters θj\theta_j are treated as a "frozen reference snapshot." In practice, at task boundary (or periodically, in the task agnostic setting) the agent samples states from its main replay buffer, it checks if the frozen network's decision is stable (via target network agreement) and if so, saves the state, the chosen action, and the action gap into a smaller, dedicated auxiliary memory which is used to compute the GAC.

3a- Can the authors report end-to-end wall-clock time per fixed number of environment steps for DQN, DQN with Value Cloning, and DQN with GAC under matched settings, separated into environment interaction, gradient updates, and auxiliary-memory construction? 3b- Memory accounting. What exactly is stored per auxiliary record for GAC and for Value Cloning, in bytes? In particular, does GAC retain the reference parameters θj\theta_j, or only the precomputed triples {s,a^j⋆(s),Mj(s)}\{s, \hat{a}^{\star}_j(s), M_j(s)\}? Does Value Cloning store the full reference action-value vector? 3c- Budget matching. The memory-scaling experiment varies a shared ``memory size'' of 100 and 10{,}000. Is this budget matched in number of retained states or in bytes? If the former, the two methods may have unequal memory footprints, which should be stated when interpreting the comparison. 3d- Reference freshness in the task-agnostic setting. Under periodic auxiliary-memory updates, whose greedy action defines a newly admitted record: the current online network, its lagged target, or the last task-boundary snapshot θj\theta_j? Clarifying this is necessary to interpret which decisions are being consolidated when updates are not aligned with task boundaries.

Questions for authors

integrated in weaknesses.

3 comments

We thank the reviewer for the careful assessment and constructive suggestions. We address the theoretical clarifications and implementation concerns below.

Weakness 1. Clarification on the reference distribution μj\mu_j.

Reference Distribution. We thank the reviewer for identifying this ambiguity. As correctly noted, the reference distribution μj\mu_j is held fixed with respect to the current parameters θk\theta_k, rather than assumed to be statistically independent of the state distributions encountered in subsequent tasks. Our formulation imposes no restriction on state-space overlap, which is fully compatible with the shared state representation space established in Section 3.

Revision. We have revised Section 4.1 to explicitly state that μj\mu_j is associated with the historical reference θj\theta_j and held fixed as the current parameters θk\theta_k evolve.

Weakness 2. Clarification on the Continual Action Gap formulation.

We thank the reviewer for this helpful suggestion. Unlike binary matching between the historical and current greedy actions, which only indicates whether the historical decision is preserved, the continual action gap quantifies its signed advantage over the strongest competing action under the current QQ-function. By capturing the margin separating the historical greedy action from its strongest competitor, rather than merely their ordering, this formulation provides a graded surrogate for gradient-based hinge regularization, extending the classical action gap to the continual learning setting.

Revision. We have clarified this distinction and its motivation in Section 4.1, emphasizing the role of the continual action gap as a decision margin for preserving historical greedy actions.

Weakness 3a. Can the authors report end-to-end wall-clock time per fixed number of environment steps for DQN, DQN with Value Cloning, and DQN with GAC under matched settings, separated into environment interaction, gradient updates, and auxiliary-memory construction?

Wall-Clock Time. We additionally report the wall-clock time over 9M environment steps on MinAtar using NVIDIA RTX 3090 GPUs. For each method, three independently seeded runs were executed concurrently on a single GPU, with wall-clock time measured separately for each process and averaged across the three runs. The measurements distinguish environment interaction, gradient updates and auxiliary memory construction.

MethodEnvironment InteractionGradient UpdatesAuxiliary MemoryTotal
DQN2.80 h17.37 hN/A24.57 h
DQN w/ Value Cloning2.80 h24.08 h10.35 s32.17 h
DQN w/ GAC2.76 h25.34 h6.34 s33.30 h
DQN w/ FAME2.89 h18.41 hN/A29.72 h
DQN w/ EWC2.74 h32.80 hN/A40.39 h

Auxiliary memory construction incurs only seconds of overhead throughout training, as updates occur exclusively at task transitions and lazy reservoir sampling avoids computing auxiliary targets for every candidate state.

Weakness 3b. Memory accounting.

Stored Records and Memory Footprint. On MinAtar, GAC stores only precomputed triples {s,a^j⋆(s),Mj(s)}\{s,\hat a_j^\star(s),M_j(s)\}, without historical network parameters. Value Cloning instead stores the full reference QQ-value vector per state. The actual footprints for 10,000 records are:

Auxiliary Memory ComponentGACValue Cloning
States26.7028 MiB26.7028 MiB
Historical Greedy Actions78.125 KiBN/A
Historical Action Gaps39.0625 KiBN/A
Reference Q-valuesN/A234.375 KiB
Total26.8173 MiB26.9318 MiB

Weakness 3c. Budget matching.

Memory Budget Matching. Our memory-scaling experiments vary capacity from 100 to 10,000 records, matching record counts rather than byte budgets. At equal capacity, footprints differ because GAC stores a historical action and gap, whereas Value Cloning stores the full reference QQ-vector. Value Cloning therefore uses slightly more memory, as quantified above. The experiments thus compare matched record capacities, not strictly matched byte budgets.

Revision. Section 5.2 now clarifies that memory capacity is matched by record count, not byte-level storage.

Weakness 3d. Reference freshness in the task-agnostic setting.

In task-boundary free Forager, auxiliary memory is updated every 50K environment steps, independently of task transitions. The online network determines each candidate's historical greedy action, while the lagged target network verifies greedy-action alignment. Upon agreement, GAC stores the shared action and the smaller of their two action gaps, which remain fixed until replacement. Thus, consolidated decisions reflect the online policy at successive memory updates rather than a frozen task-boundary snapshot.

Revision. We have clarified this procedure in Appendix B, where Algorithms 3 and 4 detail the complete auxiliary memory sampling and update process.

Reviewer_H1PbOpenReview ↗

Resolved my concerns

Thank you for your clarifications and for updating manuscript and additional material. I have adjusted my rating and would argue for accepting this paper.

Thank you for taking the time to review our revisions and additional material. We are grateful for your consideration and support of our work.

Reviewer_ZzXrOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Greedy Action Consolidation (GAC) for reducing catastrophic forgetting in value-based continual reinforcement learning. GAC encourages previously greedy actions to retain a fraction of their historical advantage, while allowing Q-values to change. The authors provide theoretical results connecting the regularizer to behavioral forgetting and historical return, and evaluate the method with DQN and PT-DQN on MinAtar and Forager.

Critical strengths
  • The method has a clear motivation: retaining a greedy policy does not require preserving every Q-value. GAC focuses on maintaining the advantage of previously selected actions, leaving more freedom for value estimates to change during subsequent learning. This is a sensible approach to protecting past behavior, and the regularizer is straightforward to incorporate into existing value-based agents.

  • The paper supports the method with both theoretical analysis and encouraging empirical results. The proofs appear sound under the stated assumptions. On MinAtar, GAC achieves substantially higher lifetime returns than EWC and both FAME learners. It also improves over DQN on Forager, although Online EWC achieves a slightly higher average-reward AUC. The ablations of the retention coefficient and memory configuration help explain the method’s behavior.

Critical weaknesses
  • The auxiliary memory procedure needs a clearer description. The paper provides memory sizes and update intervals, but I could not determine exactly how states are selected, which entries are replaced when the memory is full, or whether stored action labels and margins are later recomputed. These choices determine which past decisions GAC protects and could affect its behavior considerably. More detail would help both reproducibility and interpretation of the results.

  • The evaluation is somewhat narrow. It covers three MinAtar games and one Forager environment configuration. The MinAtar sequences are also cyclic shifts of the same ordering, preserving the same transitions between games. These experiments provide useful evidence, but leave uncertainty about whether the gains extend to a broader range of tasks and more demanding visual environments. Including additional MinAtar games or a small continual Atari experiment would strengthen the empirical case.

Questions for authors
  • Could you describe the auxiliary memory update procedure in more detail? In particular, where are candidate states sampled from, how are entries selected for replacement, and do stored action labels and margins remain fixed? Please clarify this for both the task-boundary updates in MinAtar and the periodic updates in Forager.

  • Could the authors explain why the MinAtar evaluation is limited to three games, excluding Asterix and Seaquest? Also, were experiments on full Atari considered, and were there practical constraints that prevented this evaluation?

3 comments

We thank the reviewer for recognizing GAC's motivation, theoretical foundations, and empirical results. We address the concerns below.

Weakness 1 & Question 1. The auxiliary memory procedure needs a clearer description.

Auxiliary Memory Construction and Replacement. Both methods update auxiliary memory via reservoir sampling, at task switches in task-boundary-aware settings and every 50K steps otherwise. Candidates are drawn from the current replay buffer, with lazy reservoir sampling assigning replacement slots. Value Cloning stores historical QQ-value vectors. GAC additionally retains only states for which online and target greedy actions agree, storing the shared action and the smaller action gap to limit sensitivity to overestimation. Misaligned candidates are discarded without resampling, so GAC uses filtered rather than strictly uniform reservoir sampling.

Revision. Appendix B now provides the complete sampling and replacement procedures, with pseudocode for both methods.

Weakness 2 & Question 2. The evaluation is somewhat narrow.

Our Breakout, Space Invaders, and Freeway selection follows established continual MinAtar protocols for PT-DQN [1] and FAME [2], ensuring comparability. We extend each task phase from 0.5M to 1M steps and total training from approximately 3.5M to 9M steps, enabling a more comprehensive assessment of forgetting and reacquisition across repeated task exposures.

Randomized Task Sequences. To vary transitions beyond cyclic shifts, we additionally test two randomly generated sequences, each containing every game once per three-phase cycle. This preserves balanced task exposure and the original training budget. Each sequence is evaluated over three independent runs with different random seeds.

MethodBreakoutSpace InvadersFreeway
DQN4.672 ± 0.44725.303 ± 1.6597.264 ± 2.410
DQN w/ EWC6.23 ± 0.42937.488 ± 1.1299.630 ± 0.912
DQN w/ GAC14.037 ± 1.90078.421 ± 8.26131.223 ± 3.373
DQN w/ Value Cloning14.780 ± 1.27583.958 ± 10.47916.017 ± 1.972
DQN w/ FAME (fast learner)4.275 ± 0.56726.223 ± 3.70211.268 ± 1.557
DQN w/ FAME (meta learner)11.161 ± 2.12112.054 ± 1.7198.915 ± 4.224
PT-DQN5.107 ± 0.92420.178 ± 2.6817.366 ± 7.260
PT-DQN w/ GAC18.608 ± 5.62231.005 ± 7.50112.245 ± 11.425

Results are consistent with our original findings. GAC improves over DQN on all games and enhances PT-DQN across tasks. Value Cloning retains its advantage on Space Invaders, while GAC leads on Freeway and PT-DQN with GAC on Breakout, demonstrating benefits beyond the original cyclic ordering.

Additional MinAtar Games. We also evaluate a nine-phase sequence repeating Seaquest, Asterix, and Space Invaders three times under the original configuration. As reported in the newly added Appendix D, GAC exhibits broadly comparable performance to Value Cloning on Seaquest and Asterix, while Value Cloning retains its advantage on Space Invaders, consistent with our original findings. These results provide further empirical support for the theoretical motivation of GAC, demonstrating its ability to mitigate forgetting across additional MinAtar tasks while maintaining competitive performance.

Additional Forager Evaluation. We evaluate the more challenging unending four-biome Forager environment introduced in [3], where task transitions are dynamically driven by the agent's own behavior. The table reports mean normalized AUC over 15 independent runs with different random seeds.

MethodNorm. AUC
DQN w/ GAC0.097 ± 0.014
DQN w/ Online EWC0.089 ± 0.016
DQN0.061 ± 0.020
PT-DQN w/ GAC0.045 ± 0.021
DQN w/ Value Cloning0.010 ± 0.005
PT-DQN0.004 ± 0.002

GAC achieves the highest normalized AUC, outperforming Online EWC and extending the evidence beyond the original two-biome setting.

Full Atari. We agree that full-Atari continual learning experiments would strengthen the empirical case. However, controlled evaluations across multiple pixel-based tasks require substantially greater computation under our extended horizons and repeated task exposures, placing them beyond the rebuttal scope. We leave them for future work.

Revision. We have incorporated the unending four-biome Forager results into Section 5.2 and provided the additional MinAtar experiments, including task-diversity and randomized sequence evaluations, in Appendix D.

References

[1] Anand et al. Prediction and Control in Continual Reinforcement Learning. NeurIPS, 2023.

[2] Sun et al. Principled Fast and Meta Knowledge Learners for Continual Reinforcement Learning. ICLR, 2026.

[3] Tang et al. Forager: A Lightweight Testbed for Continual Learning with Partial Observability in Reinforcement Learning. RLC, 2026.

I would like to thank the authors for their rebuttal. I am satisfied with the clarifications and new results provided, so I will maintain my positive assessment and wish the authors good luck with the paper!

Thank you for your careful review and consideration of our clarifications and additional results. We are grateful for your continued support and positive assessment of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.