- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
To address catastrophic forgetting in CRL and the overly restrictive nature of conventional value cloning, this manuscript proposes GAC from the perspective of decision preservation rather than value preservation. By defining the continual action gap and constructing a hinge regularization loss based on historical action gaps, the manuscript relaxes constraints on the absolute magnitudes of Q values while maintaining the relative advantage of historical greedy actions. The manuscript provides relevant theoretical analysis and validates the effectiveness of the method on the MinAtar and Forager environments.
Critical strengths
- The manuscript introduces a reasonable and practically meaningful research entry point. It explicitly distinguishes numerical prediction stability from behavioral stability, offering intuitive observations that provide compelling motivation for the proposed method.
- The method design is concise and intuitive. It does not require modifying the core objective of base TD learning, exhibiting good modular potential. The experiments also adequately validate its integration capability with DQN and PT-DQN.
- The theoretical analysis demonstrates good completeness, providing a formal explanation for the design motivation with an overall clear derivation process.
- The experimental scenarios cover both settings where task boundaries are known and settings where task boundaries are unobserved. Valuable ablation studies are conducted around the retention coefficient, memory capacity, and action alignment mechanism.
Critical weaknesses
- The manuscript lacks direct policy preservation and action ranking baselines, which in my view leaves the core contribution insufficiently clear. Although the manuscript compares against Value Cloning with the policy cloning term removed, this only represents a numerical value retention baseline and cannot represent methods that directly preserve historical policies.
- In Forager, partial observability causes perceptual aliasing and periodic goal conflicts, where the same observation may correspond to opposite optimal actions across different phases. In my assessment, the experimental design and evaluation protocol in the manuscript do not sufficiently demonstrate that the method genuinely mitigates catastrophic forgetting on historical tasks.
- The theoretical results rely on relatively strong conditions, and their guarantees for practical continual learning performance remain rather limited in my view.
- In RL, historical greedy actions are not necessarily optimal actions. Subsequent training that corrects earlier erroneous action orderings should ideally serve as positive knowledge updating, yet under the Behavioral Forgetting metric defined in the manuscript, this is counted as behavioral forgetting. Furthermore, due to Q value overestimation or training instability, erroneous actions may also produce large historical action gaps. Applying stronger consolidation to them could inadvertently entrench suboptimal policies and induce negative transfer. Therefore, I consider both the Behavioral Forgetting metric and the historical gap weighting mechanism to be fragile.
- The empirical gains in the current manuscript are inconsistent, and the verification of statistical reliability and generalization remains insufficient. For example, on Space Invaders in MinAtar, DQN+GAC underperforms Value Cloning and exhibits noticeably larger variance, while on Forager, DQN+GAC also lags behind Online EWC. In addition, I find that the random seeds, experimental scale, and statistical testing are relatively weak.
- The manuscript does not clearly explain the memory replacement strategy or how conflicting historical constraints for the same state are handled. Furthermore, I note that the manuscript lacks fair comparisons under fixed memory byte budgets and computational time budgets.
Questions for authors
- Regarding my first weakness, could the authors include direct behavioral preservation baselines to clarify the actual empirical gain brought by the specific numerical values of historical action gaps?
- Regarding my second weakness, in Forager or environments with conflicting goals, how exactly does GAC handle contradictory action gap constraints produced for the same observation across different phases? Could the authors provide controlled historical task reevaluation or recovery speed metrics?
- Regarding my third and fourth weaknesses, could the authors report the empirical distributions of historical action gaps and actual GAC loss values during real training to illustrate whether Theorem 1 is tight on empirical data?
- Regarding my fifth and sixth weaknesses, could the authors clarify the specific replacement strategy for the auxiliary memory and provide the actual memory footprint alongside training runtime overhead?
4 comments
Weakness 3,4 & Question 3. relatively strong conditions...historical greedy actions are not necessarily optimal actions...report empirical historical action gap distributions and actual GAC losses to assess whether Theorem 1 is tight on empirical data?
Theoretical Assumptions. Theorem 1 relies on a reference distribution matching the historical policy's discounted state occupancy to establish the return discrepancy bound. This is a theoretical assumption, not a requirement of practical GAC optimization. In model-free online continual RL, exact historical occupancy is generally unavailable, though approximable using retained data. GAC therefore uses auxiliary memory states sampled from the replay buffer without requiring them to follow this occupancy. We do not claim an unconditional return guarantee for arbitrary replay distributions.
Historical Action Reliability. Section 3 clarifies that denotes a learned greedy action, not necessarily an optimal one. Since optimal action labels are generally unavailable in model-free RL, GAC uses the learned greedy policy as a reference for behavioral retention. Accordingly, behavioral forgetting measures deviations from historical learned behavior, not necessarily harmful performance degradation, and may count beneficial corrections. Greedy action alignment requires agreement between online and lagged target networks and retains the smaller of their action gaps. This filtering reduces sensitivity to transient action-ranking instability and potentially overestimated gaps, although cannot guarantee optimality.
Empirical Validation. We add empirical analyses of Theorem 1. First, historical action gap distributions across Breakout, Space Invaders, and Freeway reveal task-dependent variation. Second, using actual auxiliary memory states, we track behavioral forgetting and its empirical upper bound throughout training. This reflects the practical regularizer's state distribution rather than assuming theoretical historical occupancy. The bound remains around –, while observed behavioral forgetting stays within approximately –. Though conservative rather than numerically tight, it provides a non-vacuous characterization under the practical distribution.
We also report actual GAC losses from auxiliary minibatches used in MinAtar gradient updates, characterizing the magnitude and dispersion of violations.
| Logged Updates | Mean | Std. | Median | P95 | Max |
|---|---|---|---|---|---|
| 6,723 | 9.4301e-4 | 1.0420e-3 | 6.9456e-4 | 2.8211e-3 | 1.1462e-2 |
Revision. Added to Section 5.3.
Weakness 5. The empirical gains in the current manuscript are inconsistent...the verification of statistical reliability and generalization remains insufficient.
Additional Generalization Evidence. See our response to Reviewer ZzXr, Weakness 2, for additional MinAtar sequences and the challenging unending four-biome Forager evaluation. GAC achieves an average normalized AUC of over 15 independent runs in this environment, compared with for Online EWC.
Furthermore, we extend GAC to hierarchical value-based learning through a Deep Option Network (DON) based on the options framework and intra-option value learning of [2]. GAC is applied to both the option-value function and intra-option action-value functions .
| Method | Norm. AUC |
|---|---|
| DON | 0.022±0.110 |
| DON w/ GAC | 0.225±0.381 |
These preliminary results suggest that GAC can benefit hierarchical value learning by consolidating decisions across multiple levels of temporal abstraction.
Revision. We have added the unending four-biome Forager results to Section 5.2, additional MinAtar evaluations to Appendix D, and DON experiments to Appendix F.
Weakness 6. ...the memory replacement strategy or how conflicting historical constraints for the same state are handled ... lacks fair comparisons under fixed memory byte budgets and computational time budgets.
Auxiliary Memory Management. For memory construction and replacement procedures, see our response to Reviewer ZzXr, Weakness 1, and Appendix B pseudocode. Both methods use lazy reservoir sampling over replay-buffer states, while GAC additionally applies greedy action alignment.
Memory Footprint and Runtime. See our response to Reviewer H1Pb, Weaknesses 3b–3c, for per-record storage costs and clarification that memory scaling matches record counts rather than byte budgets. We also report actual Forager memory footprints.
| Method | Memory Footprint |
|---|---|
| GAC | 98.05 KiB |
| Online EWC | 161.03 KiB |
For wall-clock profiling, see our response to Reviewer H1Pb, Weakness 3a, for MinAtar measurements under matched environment-step budgets. Forager exhibits the same relative wall-clock ordering.
References
[2] Sutton et al. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 1999.
Thanks for your thorough and candid response. The new results are helpful. Your clarifications also resolve my questions about the auxiliary memory mechanism and experimental fairness. Some conceptual limitations still remain, such as conflicting constraints under partial observability and the risk of locking in suboptimal historical actions. Even so, I appreciate your honest discussion of these boundaries and your solid empirical work. I will raise my rating.
Thank you for taking the time to review our rebuttal and reconsider your assessment of our work. We sincerely appreciate your thoughtful feedback and careful consideration.
We thank the reviewer for the thoughtful and technically substantive feedback. We address the concerns point by point below.
We would like to clarify that our evaluation already includes FAME [1], which provides a principled policy-level preservation baseline for value-based continual reinforcement learning.
Policy Preservation through FAME. FAME transforms each task-specific Q-function into a Boltzmann policy,
\frac{\exp(Q_i(s,a)/\tau)} {\sum_{a'}\exp(Q_i(s,a')/\tau)},$$ and consolidates historical policies by minimizing their cumulative KL divergence from a shared meta policy. Its value-based formulation can be expressed as $$Q_k^M = \arg\min_{\tilde Q_k^M} \sum_{i=1}^{k}\sum_s \mu_i^{Q_i}(s) \left[ \mathrm{KL} \left( \pi^{Q_i}(\cdot\mid s) \Vert \tilde{\pi}_k^M(\cdot\mid s) \right) \right].$$ Here, $\mu_i^{Q_i}(s)$ weights states according to the distribution associated with the historical task-specific Q-function, while $\tilde{\pi}_k^M$ denotes the Boltzmann policy induced by the candidate meta Q-function $\tilde Q_k^M$. Thus, FAME explicitly consolidates historical action distributions through policy distillation, rather than directly preserving numerical action gaps or value magnitudes as Value Cloning does. Our existing MinAtar comparison already includes both FAME components, since its meta learner and fast learner respectively serve knowledge integration and task adaptation. As reported in Table 1, DQN with GAC achieves higher normalized lifetime AUC than both FAME components across all three environments. |Method|Breakout|Space Invaders|Freeway| |-|-:|-:|-:| |DQN w/ GAC|**14.858 ± 0.365**|**88.282 ± 19.797**| **26.121 ± 2.988** | |DQN w/ FAME (meta learner)|10.170 ± 0.271|16.798 ± 4.114|11.542 ± 3.351 | |DQN w/ FAME (fast learner)|4.737 ± 0.770|26.338 ± 1.002|11.077 ± 0.095| These results provide an empirical comparison against explicit policy-level consolidation in value-based continual reinforcement learning and demonstrate the effectiveness of directly preserving historical greedy action gaps. **Contribution of Historical Action Gap Magnitudes.** Our retention-coefficient ablation in Figure 4a further examines the contribution of the specific numerical values of historical action gaps. In particular, $\rho=0$ only penalizes negative continual action gaps, whereas $\rho=0.5$ additionally enforces the retention of at least half of each historical action gap. The consistently higher normalized AUC achieved with $\rho=0.5$ across all three MinAtar environments provides complementary evidence that preserving the specific numerical values of historical action gaps offers additional benefits. > **Weakness 2 & Question 2**. how exactly does GAC handle contradictory action gap constraints produced for the same observation across different phases? Could the authors provide controlled historical task reevaluation or recovery speed metrics? We thank the reviewer for highlighting this important issue. We clarify that GAC does not resolve perceptual aliasing or explicitly reconcile contradictory historical constraints. Specifically, when the same observation corresponds to different historical greedy actions $a$ and $b$, GAC may impose mutually incompatible constraints, $$Q(s,a)-Q(s,b)\geq \rho M_1(s)>0, \qquad Q(s,b)-Q(s,a)\geq \rho M_2(s)>0.$$ Nevertheless, GAC offers greater flexibility than Value Cloning. As established by Theorem 2, its feasible set strictly contains that of exact value preservation. This additional flexibility allows the $Q$-function to adapt more freely to changing reward mappings, although it does not guarantee the resolution of intrinsically conflicting decisions. Importantly, our Forager experiments are not intended to demonstrate that GAC solves state aliasing, but rather to empirically examine the practical advantage of the larger feasible set established by Theorem 2 under partial observability. **Controlled Historical-Task Reevaluation.** To examine this issue empirically, we additionally evaluate historical-task performance in the two-biome Forager environment. While training under reward mapping 1, we periodically evaluate the current agent under the conflicting reward mapping 2 and compute its normalized AUC, where higher values indicate better historical-task retention. |Method|Norm. AUC| |-|-:| |DQN w/ GAC|**−0.2707± 0.0431**| |DQN w/ Online EWC| −0.3780±0.0710| Both methods exhibit negative normalized AUC, confirming that conflicting reward mappings induce substantial interference. Nevertheless, GAC achieves better historical-task retention than Online EWC. **References** [1] Sun et al. Principled Fast and Meta Knowledge Learners for Continual Reinforcement Learning. ICLR, 2026.