acceptodds
Under review as a conference paper at ICLR 2027

Policy Improvement and Preservation in Continual Reinforcement Learning with Latent Families

Abstract

Continual reinforcement learning requires agents to improve in recurring environments while preserving performance on environments that are currently unavailable. Sharing policies enables reuse, but updates for one environment can harm others. We introduce a continuous-state framework that separates identification of latent environment families from the decision to share a policy. Learned Bellman comparisons identify families, while one-sided certificates use finite stored data to bound loss relative to retained reference policies, called anchors. When a shared update cannot be certified, policy splitting allows learning to continue. Under stated separation, sampling, coverage, and local neural-geometry assumptions, we establish high-probability guarantees for correct identification, anchor-relative preservation, and final -optimality, using at most one deployed policy module per family. The complete observation bound retains coverage, conditioning, representation-complexity, and acquisition factors. Under the stronger specialization that these factors remain uniformly controlled, the prescribed exploration schedule gives sufficient observation complexity where is the number of families, is module capacity, counts segments, and is a diagnostic family separation lower bound. Under additional compatibility and critic-accuracy conditions, all families can remain in one deployed module without splitting; matched constructions show when separate policies are necessary. This module reduction does not imply lower observation or total-memory cost. In a recurrent experiment with learned neural evaluators and Bellman regressions, the method matches independent policies' final performance using two modules instead of three and substantially reduces forgetting relative to ungated sharing.

Primary area:
reinforcement learning
Keywords:
continual reinforcement learning, lifelong reinforcement learning, reinforcement learning theory, sample complexity, neural function approximation, latent task identification, policy preservation, policy sharing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NRUfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a continual reinforcement learning framework for continuous state environments where KK latent families recur across SS segments with observed boundaries. The goal is to reach ε\varepsilon optimal return on every family while preserving performance on currently unavailable families when up to hh records share a deployed policy module. To balance active improvement with inactive family protection, the method separates family identification from the decision to share a policy. When a segment begins, the learner routes it using learned Bellman comparisons against stored diagnostic references, creating a new record if no score falls within Δ0/4\Delta_0/4 for a supplied separation lower bound Δ0\Delta_0. For policy improvement, the learner evaluates the incumbent policy on the active environment and proposes mixture updates πλ\pi_\lambda toward an exploratory greedy endpoint. Before updating a shared module, the algorithm uses stored reserve blocks to compute one sided finite data certificates that bound return loss relative to each protected record's frozen anchor policy and snapshot. When no sufficiently large shared update can be certified, exact copy splitting isolates the active record into a new module and takes the full update λ=1\lambda = 1. Under stated separation, coverage, and local neural geometry assumptions, the authors provide sufficient observation complexity analysis and empirical validation on control tasks.

Critical strengths

The main strength of the paper is its clean separation of family identification, anchored preservation, and progress via exact copy splitting. In continual reinforcement learning with shared policies, checking each update only against the immediately preceding policy allows small losses on inactive families to accumulate across many intervening segments. By instead certifying shared updates against a fixed anchor policy and snapshot that remain frozen between a family's own updates, the analysis charges dormant loss only once per update interval and avoids an extra factor of the segment count SS or update quota JJ in the final optimality gap. In addition, using a one sided Bellman comparison permits shared updates that improve both active and inactive families, whereas a symmetric gate would reject mutually beneficial steps, and exact copy splitting ensures that the active family can always make progress when sharing cannot be certified.

Critical weaknesses

The main concern is that the paper does not show a meaningful improvement over the naive baseline. The paper motivates policy sharing by knowledge reuse, but neither its theory nor its experiments demonstrate a benefit over its own Independent baseline, which allocates one module per inferred record (lines 436 to 437), beyond fewer deployed modules. Because routing is separated from policy sharing, setting the module capacity to h=1h = 1 recovers this Independent baseline as a special case of Algorithm 1. It already satisfies all of Theorem 1's guarantees with at most KK modules, while achieving zero dormant anchor loss, taking full steps λ=1\lambda = 1 at every own update, and requiring no reserve blocks. In fact, in Corollary 2, sharing with h>1h > 1 worsens the sufficient observation complexity bound from O(K/ε2+(S+K)/Δ02)O(K/\varepsilon^2 + (S+K)/\Delta_0^2) by adding an O(Kh/ε3)O(Kh/\varepsilon^3) preservation reserve cost under an exploratory logging floor. The only theoretical sharing benefit, Theorem 3(a), concerns deployed module count alone and follows almost directly from Assumption 4: because the active family's greedy action is assumed to simultaneously maximize every protected anchor's action value with a positive margin, the policy improvement theorem already makes the full step λ=1\lambda = 1 harmless at the population level. Certifying this in finite samples under Calibration S even requires tighter active critic accuracy of order ε3/2\varepsilon^{3/2}, raising the active critic sample budget from O(K/ε2)O(K/\varepsilon^2) to O(K/ε3)O(K/\varepsilon^3) in Corollary 43. Empirically, the recurrent control results (Tables 5 and 6) show the same pattern: independent policies match the Full method's final mean return (2.256) and worst family return (1.685), learn faster because no interaction budget is spent collecting reserve blocks (AUC 2.163 vs 2.107), and use far less retained peak storage (1.41 KiB vs 24.89 KiB) since the Full method still stores references, snapshots, and reserves for every record. The supporting CartPole study also omits the Independent baseline, leaving no setting where sharing improves return, sample efficiency, or total memory.

Questions for authors

Please see weakness

3 comments

We thank the reviewer for a careful and accurate reading; the summary of strengths matches how we see the paper. We agree with the factual premises of the weakness. The empirical points (Tables 5–6, CartPole) are addressed with new experiments in the general response; below we address the claim and the theory.

1. What the paper claims. As stated in the general response, the claim is a reduction in distinct deployed policies without giving up ε\varepsilon-optimality or protection of inactive families. Table 5 shows why this is not trivial: ungated sharing deploys one module but drops the worst family to 0.503, with mean maximum anchor loss 1.181, and certified sharing without splitting stalls at a final mean return of 1.818. On the reported stream, only the full method (neural, and its linear-evaluator reference running the same outer algorithm) matches Independent's final and worst-family return (2.256 / 1.685) with fewer modules, with anchor loss 0.009 and no run above the 0.10 tolerance. Shared updates transfer during learning (12 of the 13 audited shared installations improve both the active family and a protected family by more than 0.02, App. E.1.4), but in this task Full learns more slowly than Independent; general response (c) traces this to reserve acquisition. We agree that the abstract's phrase "Sharing policies enables reuse" suggests more than the submission demonstrates, and the revision replaces it. The CartPole comparison in general response (a) shows a setting where sharing does improve return under a matched budget.

2. h=1h=1 and Corollary 2. We agree that h=1h=1 recovers Independent and satisfies Theorem 1. This is by design: Algorithm 1 spans independent policies (h=1h=1) to maximal sharing (h=Kh=K), and Theorem 1's identification, preservation and accuracy conclusions hold for every hh. Reducing the number of deployed policies therefore never weakens the ε\varepsilon-guarantee; its price is the explicit K(h−1)K(h-1) reserve term. We also agree that the bound is never smaller for h>1h>1. How much larger depends on the logger: the ε−3\varepsilon^{-3} rate arises from an exploratory logger with plog⁡=Θ(ε)p_{\log}=\Theta(\varepsilon), whereas a fixed-floor logger gives O~(Khε−2+(S+K)Δ0−2)\tilde O(Kh\varepsilon^{-2}+(S+K)\Delta_0^{-2}) (Corollary 2), a factor hh at the same ε\varepsilon-order as independent policies. The revision states this tradeoff after Corollary 2.

3. Theorem 3. We agree that at the population level, Assumption 4 makes the full step harmless; this is the policy-improvement step, and the revision says so before Theorem 3. The theorem's content lies elsewhere. (a) Inactive families cannot be queried, so the full step must be certified from finite stored reserves and learned snapshots; Calibration S is what makes this one-sided finite-sample certificate accept λ=1\lambda=1 at every own slot. Its ε3/2\varepsilon^{3/2} critic precision is sufficient for our proof, and we do not claim it is necessary, as stated after Theorem 3. (b) Part (b) does not follow from Assumption 4. On the conflicting instance, every common stationary policy has gap at least LconfL_{\mathrm{conf}} on some family, so separate policies are necessary. The same algorithm with common supplied budgets uses one module on the compatible instance and two on the conflicting one, without testing compatibility, whereas independent policies use two on both. Theorem 3 thus gives a sufficient condition for a single deployed policy and a construction in which separate policies are necessary, and the algorithm attains the minimum module count on both instances of that construction.

Dear Reviewer NRUf,

Thank you again for your careful review. As the discussion period ends on [date], we wanted to check whether our response addresses your concerns. We have uploaded a revised PDF, with all changes marked in blue, that adds a CartPole comparison with independent policies (App. E.3), scaling to twelve families (App. E.4.2), and a diagnostic identifying the source of the learning-speed gap (App. E.4.3). It also states the module-capacity tradeoff after Corollary 2 and the population-level remark before Theorem 3.

We would be grateful for any further comments or questions, and we are happy to clarify anything that remains unclear.

Best regards, The authors

Thank you for the response and additional results. I appreciate the clarification that the main benefit is reducing the number of deployed policies. However, my central concern about the value of sharing over Independent remains unresolved. The technical presentation also remains difficult to follow. Overall, the rebuttal has not sufficiently addressed my concerns, and I will keep my score unchanged.

Reviewer_x9MYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies recurring latent environments with observed switching boundaries. It combines Bellman-based identification, anchor-relative preservation certificates, and policy splitting, with guarantees for identification, preservation, final accuracy, and observation cost.

Critical strengths

I find the separation of task identity from policy compatibility to be a useful idea. The framework provides joint high-probability guarantees for preservation and learning progress, even though inactive environments supply no fresh data. In addition, it guarantees single-module sharing under stronger compatibility conditions.

Critical weaknesses

This area lies outside my core expertise, and some of my comments may reflect limited familiarity with the related literature.

  1. The theoretical novelty would benefit from clearer differentiation. The paper explains the roles of fixed anchors and variable-step propagation, but the technical advance they offer over existing analyses remains unclear to me. Appendix F offers qualitative comparisons; a quantitative comparison of sample complexity and performance guarantees would help establish the contribution.

  2. The practical advantages over existing methods remain to be established. The experiments compare internal variants and independent policies, without established continual-RL baselines. The method matches the final return of independent policies with fewer modules, but independent policies show faster learning and use less retained numeric storage. The CartPole study includes no baseline method for comparison.

Questions for authors
  1. Could the authors clarify which proof ingredient is new, and how the guarantees compare with prior results in quantitative terms, accounting for differences in assumptions?
  2. Would it be possible to compare performance with established continual-RL methods, in order to demonstrate the practical benefits?
3 comments

Response to Reviewer x9MY

We thank the reviewer for the clear questions. The empirical weakness (faster learning and lower storage of independent policies, and the CartPole baseline) is addressed in general response (a)–(c), and Question 2 in general response (e) and App. E.4.4.

Q1. Which proof ingredients are new, and how the guarantees compare. Three ingredients are new relative to the works in App. F, and each removes a specific cost.

  1. Fixed-anchor comparison. A natural alternative certifies each shared update against the policy immediately before it, as a repeated safe-improvement test would. A dormant family's loss is then bounded by the sum of the per-update allowances, so the bound grows with the number of installations made while the family is inactive, up to the number of segments SS. Our certificates compare every installation with the family's fixed anchor, so the preservation bound in eq. (8) charges the loss once per update interval: a coverage factor times the sum of the preservation tolerance and the snapshot error, divided by 1−γ1-\gamma, with no factor of SS.
  2. Variable-step transport. Across the JJ own updates of a family, the accumulated critic and interference errors are propagated through contractive improvement operators whose step weights are bounded below by the minimum step size. This gives the interference term in eq. (9), which scales with the preservation tolerance and snapshot error divided by the squared effective horizon and the minimum step size, without an additional factor JJ.
  3. Finite reserves with controlled disclosure, and exact-copy splitting. Reserves make the one-sided concentration valid for comparisons chosen adaptively from the visible history, at the explicit cost K(h−1)plog⁡−1ε−2K(h-1)p_{\log}^{-1}\varepsilon^{-2} in Corollary 2. Splitting guarantees progress without any compatibility assumption, which is what allows Theorem 1 to hold for every hh.

A direct comparison of rates with prior work requires aligning the guaranteed quantity, and the closest results bound different quantities under different assumptions: regret with observed contexts and linear features (UCBlvd), PAC-MDP sample complexity with value transfer in tabular MDPs (Lipschitz lifelong RL), and identification costs with finite states (Brunskill–Li; Nguyen–Mehta). The clearest common term is identification. Brunskill–Li pay an inverse-square separation cost Γ−2\Gamma^{-2} for discovering and re-identifying tasks among finitely many tabular MDPs, and Nguyen–Mehta pay a clustering cost of order λ−2\lambda^{-2} per presentation under transition separation λ\lambda. Our (S+K)Δ0−2(S+K)\Delta_0^{-2} has the same inverse-square structure, one identification per segment plus one reference per family, for continuous states and with separation measured by diagnostic Bellman residuals rather than transition distances. UCBlvd observes contexts and so has no identification term, and Lipschitz RMax bounds per-task PAC sample complexity, O~(ns2A ϵ−3(1−γ)−3)\tilde O(n_s^2A\,\epsilon^{-3}(1-\gamma)^{-3}) in the worst case, in tabular MDPs without latent identification. App. F now includes a table (Table 16) with each work's setting, guaranteed quantity and rate, aligned with our terms where a common quantity exists.

Q2. Comparison with established continual-RL methods. General response (e) adds online EWC and CLEAR-style replay with behaviour cloning under the same routing, critics and budget, with β\beta tuned on separate seeds. Unconditioned variants are bounded by the exact single-policy ceiling of 1.067 and reach at most 0.759 in worst-family return. The strongest baseline, record-conditioned replay, is statistically indistinguishable from Full in worst-family return but exceeds the anchor-loss tolerance in 3 of 10 runs, against none for Full. Full therefore matches the strongest established method empirically while providing the finite-sample preservation and progress guarantees those methods lack.

Dear Reviewer x9MY,

Thank you again for your review and your questions. As the discussion period ends on [date], we wanted to check whether our response addresses your concerns. We have uploaded a revised PDF, with all changes marked in blue, that adds comparisons with online EWC and CLEAR-style replay under the same routing, critics, and budget (App. E.4.4), and a quantitative comparison of our guarantees with prior results (App. F, Table 16). It also adds a CartPole comparison with independent policies (App. E.3).

We would be grateful for any further comments or questions, and we are happy to clarify anything that remains unclear.

Best regards, The authors

Reviewer_x9MYOpenReview ↗

Response to authors

Thank you for the detailed response and for running the new experiments. I appreciate your explanation of the three proof ingredients and the addition of Table 16.

I acknowledge that your method deploys fewer policies and offers conditional preservation guarantees that neither EWC nor replay provides. Overall, though, the advantage on either the practical or the theoretical side still does not seem sufficiently clear to me, so I am keeping my current score. I agree with reviewer siYK that a complete rewrite is necessary for the resubmission.

Reviewer_siYKOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper studies continual reinforcement learning. In this setting, there are multiple environments, and the goal is to improve in recurring environments while preserving performance on environments that are currently unavailable.

In this paper, the authors studied how to solve the identification problem (i.e., which environment has returned), and how to improve policies. The authors propose a new algorithm that aims to answer these questions.

Critical strengths

The paper studies continual reinforcement learning, which is a fundamental problem that deserves to be studied in RL.

Critical weaknesses

The paper is hard to read and to understand . A lot of new terminologies were introduced without a proper definition.

To name a few places

  1. In Contribution 1, the authors wrote: Comparing dormant changes directly with fixed named anchors avoids accumulating a separate preservation loss for every intervening update. It is unclear what dormant changes mean, and fixed named anchors were also not properly defined.
  2. In the problem setup, the authors wrote The learner receives a diagnostic separation lower bound. The way this lower bound is used was not mentioned until Section 4.
  3. The authors use a two-layer network as the evaluator. However it is never motivated. Why should we use such a function class? What activation functions will be used?

These are only a few examples. Such omission are ubiquitous in the technical part of the paper.

In short, the writing quality of the paper is not satifying, and it clearly does not meet the bar of ICLR.

Questions for authors

See weaknesses above.

3 comments

We thank the reviewer for reading the technical sections closely. We agree that several terms were used before they were defined, and we have revised the presentation accordingly. We answer the three examples directly and then describe the broader changes.

1. "Dormant changes" and "fixed named anchors" (Contribution 1). A record is dormant while its family is inactive. A dormant change is a change to the policy a dormant record deploys, caused by another record's update to the module they share. A record's named anchor πˉi\bar\pi_i is its own reference policy: the policy it was first assigned, or its latest accepted own update. The anchor stays fixed while the record is dormant and is refreshed only by the record's own updates. The sentence in Contribution 1 means the following: every shared update is compared with the fixed anchor of each dormant record that shares the module, rather than with the policy immediately before the update. The loss bound for a dormant family therefore does not accumulate across the updates other families make while it is inactive (Theorem 1(ii)). Both terms are now defined at first use in Contribution 1 and listed in the new notation table (App. G).

2. How the separation bound Δ0\Delta_0 is used. The learner uses Δ0\Delta_0 only for routing. When a segment begins, a stored record matches it if that record's learned diagnostic Bellman residual on the active data is at most Δ0/4\Delta_0/4 (eq. (4)); if no record matches, a new record is created. Assumption 1 requires Δ0\Delta_0 to be a lower bound on the diagnostic separation between every pair of distinct families: if the diagnostic action-value function of one family is plugged into the diagnostic Bellman operator of any other family, the resulting Bellman residual, measured in the root-mean-square norm under the second family's diagnostic sampling distribution, is at least Δ0\Delta_0 (eq. (7)). This is what lets the threshold Δ0/4\Delta_0/4 separate correct matches from incorrect ones. We have added this explanation to Section 2, where Δ0\Delta_0 is introduced.

3. Why a two-layer evaluator, and which activation. The theory requires end-to-end guarantees for trained critics without assuming an optimization oracle. Two-layer networks trained within a ball around their initialization are the standard setting in which finite-width optimization guarantees of this kind are available (Du et al., 2019, discussed in App. B.2), and our analysis derives the local geometry it needs for this class rather than assuming it. The outer algorithm uses the evaluator only through its accuracy guarantee, so the choice of architecture does not affect routing, certification or splitting. Empirically, a linear-evaluator reference produces the same policy trajectories as the neural evaluator (Table 5). The theory allows any twice continuously differentiable activation with bounded derivatives on the reachable preactivations, ∣σ′(z)∣≤L1|\sigma'(z)| \le L_1 and ∣σ′′(z)∣≤L2|\sigma''(z)| \le L_2 (App. D.3). The experiments use σ(z)=z2\sigma(z) = z^2 in the recurrent task (App. E.2.1) and tanh⁡\tanh in CartPole (App. E.3). We now state the activation class and this motivation next to eq. (3).

4. Presentation more broadly. Beyond these three examples, the uploaded revision (changes in blue) (i) adds a notation table (App. G) covering the symbols and terms of Sections 1–5; (ii) adds a short overview before Algorithm 1 that follows one segment through routing, policy improvement, certification and splitting; and (iii) states the objective of the paper explicitly in the introduction.

Dear Reviewer siYK,

Thank you again for your review. As the discussion period ends on [date], we wanted to check whether our response addresses your concerns about presentation. We have uploaded a revised PDF, with all changes marked in blue, that defines the terms you pointed out at first use (Section 1), explains how Δ0\Delta_0 is used where it is introduced (Section 2), states the activation class and the motivation for the evaluator next to eq. (3), adds a short overview before Algorithm 1, and adds a notation table (App. G).

We would be grateful for any further comments or questions, and we are happy to clarify anything that remains unclear.

Best regards, The authors

Thanks for the detailed response. After reading the rebuttal and the updated version, I believe a complete rewrite is necessary for the readability of the submission, which necessitates another round of peer-review. The paper in its shape is not ready to be published at ICLR.

In particular, the three places mentioned in my initial review just gave only a few examples. Such omissions are ubiquitous in the technical part of the paper. Even after the revision, the paragraph started with ``Learned functions and sampling laws'' is still extremely hard to understand, and more explanations must be added to justify why such a two-layer NN should be used for function approximation.

Given this, I will keep weak rejection as my final recommendation.

Comments on the paper

2 comments

We thank the three reviewers for their careful reading. Two concerns are shared by Reviewers NRUf and x9MY: whether sharing offers any practical benefit over independent policies, and how the method compares with baselines. This response reports new experiments on both. Individual responses address the remaining points, including the presentation issues raised by Reviewer siYK. We have uploaded a revised PDF in which every addition is marked in blue; the main text remains within ten pages.

What the paper claims. The paper's claim is that the number of distinct deployed policies can be reduced below one per family without giving up the two properties that make independent policies attractive: ε\varepsilon-optimality on every family and bounded loss on inactive families. The submitted paper does not claim that sharing improves return, and it states that the reduction does not imply lower observation or total-memory cost. Independent policies obtain both properties trivially by never sharing; unprotected sharing reduces the policy count but loses them. The experiments below test this claim at larger scale, against an Independent baseline in CartPole, and against established continual-RL methods.

(a) CartPole with an Independent baseline (App. E.3). Each descendant receives a private head initialized from its parent's head, with the same 64-episode active-return screen and mixture weight as Full. Both arms start from the same verified checkpoint at the beginning of the descendant phase.

Five seedsFull (shared heads)Independent (warm start)
Deployed heads48
Unique critic parameters55,32055,320
Installed proposals40/12040/120
Descendant gain over zero-shot, mean / min0.1658 / 0.04470.1658 / 0.0447
Parent return change during descendant phase+0.1510
Final-revisit return0.9350.852
Charged learning transitions8,097,6358,097,635

The preservation gate admitted the largest weight on all 120 attempts, so both arms install the same descendant updates. In Full, those updates also change the policy the parents deploy, raising parent returns by 0.151 on average (95% CI [0.110,0.192][0.110, 0.192]) using only descendant data; a private parent head changes only when the parent is active. Sharing therefore improves final-revisit return by 0.083 (paired 95% CI [0.050,0.117][0.050, 0.117], higher in all five seeds) under the same budget, with half the deployed policies. Part of this comes from routing: both arms miss the exact record on the same 8 of 40 revisits, which costs Independent 0.039 there and Full nothing, since siblings deploy the same head. The saving is in distinct deployed policies, not parameters: frozen mixture components are shared by reference, so both arms store the same critics.

(b) Scaling the number of families (App. E.4.2). We extend the recurrent task to K∈{4,6,8,12}K\in\{4,6,8,12\} families in two compatibility groups (minimum routing separation 0.12; cross-group common-policy gap at least 0.188 by exact computation), with ten paired seeds and the join rule in (d).

KKModules, Full [95% CI] / IndependentAUC deficit vs. Independent [95% CI]Full runs above 0.10 anchor loss
42.4 [2.0, 2.8] / 40.045 [0.037, 0.054]1/10
62.7 [2.0, 3.4] / 60.026 [0.022, 0.031]0/10
83.1 [2.4, 3.8] / 80.018 [0.014, 0.022]0/10
123.6 [3.0, 4.2] / 120.014 [0.009, 0.018]1/10

In every seed, both methods reach the exploration-constrained optimum on every family at every KK. The two runs above tolerance have losses of at most 0.117. Full's learning-speed deficit narrows from K=4K=4 to K=12K=12 by 0.032 (paired 95% CI [0.024,0.039][0.024, 0.039], in all ten seed pairs) but does not close; (c) identifies its source.

(c) Learning speed (App. E.4.3). Full spends on average 126,157 of its 983,040 charged transitions acquiring reserves. Drawing reserves from a separate stream outside the charged budget (a diagnostic using extra interactions) raises Full's AUC by 0.053 (paired 95% CI [0.049,0.057][0.049, 0.057]), from 2.107 to 2.161. The remaining difference from Independent's 2.163 is 0.002 (CI [−0.0004,0.0044][-0.0004, 0.0044]) and not significant, so the diagnostic closes 96% of the original gap of 0.055 (CI [0.053,0.058][0.053, 0.058]). The difference is the price of the reserves that protection requires, not slower improvement under sharing. Reserves also dominate Full's retained storage; we make no total-memory claim.

(d) Arrival order (App. E.4.1; Algorithm 1 and App. D.7). As submitted, Algorithm 1 assigns a new record to the first module with space, which makes the outcome order-dependent: when the conflicting family arrives first, the aligned families join its module and split, giving three modules, and 7 of 10 runs in each such order exceed the 0.10 tolerance. We replace this with a return-ranked rule: a new record evaluates each module with space on active-family data, charged to the budget, and joins the best. Across all six orders, this gives two modules in 56 of 60 runs, no run above tolerance, and Independent's final mean and worst-family return in every run; on the reported stream it makes the same joins as first-fit. Theorem 1's preservation and progress arguments carry over, since the proof never uses the first-fit order, joining leaves the chosen module's policy unchanged, and the ranking evaluations are measurable with respect to the learner's history. The ranking uses fewer than K(K−1)/2K(K-1)/2 fixed-size evaluations in total, and the chosen module's snapshot is fitted as before. These evaluations enter the theorem's call cap, active-time requirement and observation ledger, but they need no accuracy guarantee, so they do not change the ε\varepsilon-dependence of Corollary 2. Theorem 3 is unaffected, since only one module is eligible at each join in its constructions.

(e) Established continual-RL methods (App. E.4.4). On the recurrent task we added online EWC with a diagonal Fisher (Kirkpatrick et al., 2017; Schwarz et al., 2018) and CLEAR-style replay with behaviour cloning (Rolnick et al., 2019; behaviour-cloning term only, since critics are fitted as in Full). Each uses one shared policy network, either unconditioned or conditioned on the inferred record, with the same routing, critics, greedy endpoints and charged budget as Full. The replay buffer holds as many states per record as Full's reserves. The regularization weight β\beta was tuned on five separate seeds and evaluated on the ten paired seeds; Figure 6 shows the full β\beta sweep.

Method (selected β\beta)Worst-family returnMean max anchor lossRuns > 0.10AUC
Full1.6850.0090/102.107
Independent1.6850.0000/102.163
EWC, record-conditioned (0.3)1.0381.34810/101.805
Replay-BC, record-conditioned (0.1)1.5780.0733/102.085
Best unconditioned (unprotected fine-tuning)0.7590.91010/101.684

No unconditioned method can exceed 1.067, the exact worst-family return of the best single common stationary policy; the unconditioned baselines reach at most 0.759. With record conditioning, replay with behaviour cloning is the strongest baseline. Its worst-family return is not significantly different from Full's (paired difference −0.106-0.106, 95% CI [−0.283,0.071][-0.283, 0.071]), and on the evaluation seeds it reaches Full's final return at β=3\beta=3. At both the selected β\beta and β=3\beta=3, 3 of 10 runs exceed the 0.10 anchor-loss tolerance, against none for Full. Record-conditioned EWC forgets heavily at every β\beta in the grid (mean maximum anchor loss at least 1.17). A record-conditioned network deploys a distinct policy for each record, so it does not reduce the number of distinct deployed policies. Neither method provides a finite-sample bound on loss for inactive families or a mechanism that guarantees progress when a shared update conflicts.

Revisions in the uploaded PDF (in blue). (i) The abstract and introduction state the objective as reducing deployed policies under ε\varepsilon-optimality and anchor preservation. (ii) Section 5 states the hh-tradeoff and the logger dependence of Corollary 2, and adds the population-level remark before Theorem 3. (iii) Algorithm 1, App. D.7 and the observation ledger use the return-ranked join rule. (iv) Section 6 summarizes, and App. E.3 and E.4 report, the CartPole comparison, scaling, reserve diagnostic with a breakdown of interactions and storage, arrival-order results under both rules, and the established-baseline comparison. (v) The presentation changes requested by Reviewer siYK are in Sections 1–3 and the new notation table (App. G). (vi) App. F adds a quantitative comparison of guarantees (Table 16).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.