- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces a continual reinforcement learning framework for continuous state environments where latent families recur across segments with observed boundaries. The goal is to reach optimal return on every family while preserving performance on currently unavailable families when up to records share a deployed policy module. To balance active improvement with inactive family protection, the method separates family identification from the decision to share a policy. When a segment begins, the learner routes it using learned Bellman comparisons against stored diagnostic references, creating a new record if no score falls within for a supplied separation lower bound . For policy improvement, the learner evaluates the incumbent policy on the active environment and proposes mixture updates toward an exploratory greedy endpoint. Before updating a shared module, the algorithm uses stored reserve blocks to compute one sided finite data certificates that bound return loss relative to each protected record's frozen anchor policy and snapshot. When no sufficiently large shared update can be certified, exact copy splitting isolates the active record into a new module and takes the full update . Under stated separation, coverage, and local neural geometry assumptions, the authors provide sufficient observation complexity analysis and empirical validation on control tasks.
Critical strengths
The main strength of the paper is its clean separation of family identification, anchored preservation, and progress via exact copy splitting. In continual reinforcement learning with shared policies, checking each update only against the immediately preceding policy allows small losses on inactive families to accumulate across many intervening segments. By instead certifying shared updates against a fixed anchor policy and snapshot that remain frozen between a family's own updates, the analysis charges dormant loss only once per update interval and avoids an extra factor of the segment count or update quota in the final optimality gap. In addition, using a one sided Bellman comparison permits shared updates that improve both active and inactive families, whereas a symmetric gate would reject mutually beneficial steps, and exact copy splitting ensures that the active family can always make progress when sharing cannot be certified.
Critical weaknesses
The main concern is that the paper does not show a meaningful improvement over the naive baseline. The paper motivates policy sharing by knowledge reuse, but neither its theory nor its experiments demonstrate a benefit over its own Independent baseline, which allocates one module per inferred record (lines 436 to 437), beyond fewer deployed modules. Because routing is separated from policy sharing, setting the module capacity to recovers this Independent baseline as a special case of Algorithm 1. It already satisfies all of Theorem 1's guarantees with at most modules, while achieving zero dormant anchor loss, taking full steps at every own update, and requiring no reserve blocks. In fact, in Corollary 2, sharing with worsens the sufficient observation complexity bound from by adding an preservation reserve cost under an exploratory logging floor. The only theoretical sharing benefit, Theorem 3(a), concerns deployed module count alone and follows almost directly from Assumption 4: because the active family's greedy action is assumed to simultaneously maximize every protected anchor's action value with a positive margin, the policy improvement theorem already makes the full step harmless at the population level. Certifying this in finite samples under Calibration S even requires tighter active critic accuracy of order , raising the active critic sample budget from to in Corollary 43. Empirically, the recurrent control results (Tables 5 and 6) show the same pattern: independent policies match the Full method's final mean return (2.256) and worst family return (1.685), learn faster because no interaction budget is spent collecting reserve blocks (AUC 2.163 vs 2.107), and use far less retained peak storage (1.41 KiB vs 24.89 KiB) since the Full method still stores references, snapshots, and reserves for every record. The supporting CartPole study also omits the Independent baseline, leaving no setting where sharing improves return, sample efficiency, or total memory.
Questions for authors
Please see weakness
3 comments
Dear Reviewer NRUf,
Thank you again for your careful review. As the discussion period ends on [date], we wanted to check whether our response addresses your concerns. We have uploaded a revised PDF, with all changes marked in blue, that adds a CartPole comparison with independent policies (App. E.3), scaling to twelve families (App. E.4.2), and a diagnostic identifying the source of the learning-speed gap (App. E.4.3). It also states the module-capacity tradeoff after Corollary 2 and the population-level remark before Theorem 3.
We would be grateful for any further comments or questions, and we are happy to clarify anything that remains unclear.
Best regards, The authors
Thank you for the response and additional results. I appreciate the clarification that the main benefit is reducing the number of deployed policies. However, my central concern about the value of sharing over Independent remains unresolved. The technical presentation also remains difficult to follow. Overall, the rebuttal has not sufficiently addressed my concerns, and I will keep my score unchanged.
We thank the reviewer for a careful and accurate reading; the summary of strengths matches how we see the paper. We agree with the factual premises of the weakness. The empirical points (Tables 5–6, CartPole) are addressed with new experiments in the general response; below we address the claim and the theory.
1. What the paper claims. As stated in the general response, the claim is a reduction in distinct deployed policies without giving up ε-optimality or protection of inactive families. Table 5 shows why this is not trivial: ungated sharing deploys one module but drops the worst family to 0.503, with mean maximum anchor loss 1.181, and certified sharing without splitting stalls at a final mean return of 1.818. On the reported stream, only the full method (neural, and its linear-evaluator reference running the same outer algorithm) matches Independent's final and worst-family return (2.256 / 1.685) with fewer modules, with anchor loss 0.009 and no run above the 0.10 tolerance. Shared updates transfer during learning (12 of the 13 audited shared installations improve both the active family and a protected family by more than 0.02, App. E.1.4), but in this task Full learns more slowly than Independent; general response (c) traces this to reserve acquisition. We agree that the abstract's phrase "Sharing policies enables reuse" suggests more than the submission demonstrates, and the revision replaces it. The CartPole comparison in general response (a) shows a setting where sharing does improve return under a matched budget.
2. h=1 and Corollary 2. We agree that h=1 recovers Independent and satisfies Theorem 1. This is by design: Algorithm 1 spans independent policies (h=1) to maximal sharing (h=K), and Theorem 1's identification, preservation and accuracy conclusions hold for every h. Reducing the number of deployed policies therefore never weakens the ε-guarantee; its price is the explicit K(h−1) reserve term. We also agree that the bound is never smaller for h>1. How much larger depends on the logger: the ε−3 rate arises from an exploratory logger with plog=Θ(ε), whereas a fixed-floor logger gives O~(Khε−2+(S+K)Δ0−2) (Corollary 2), a factor h at the same ε-order as independent policies. The revision states this tradeoff after Corollary 2.
3. Theorem 3. We agree that at the population level, Assumption 4 makes the full step harmless; this is the policy-improvement step, and the revision says so before Theorem 3. The theorem's content lies elsewhere. (a) Inactive families cannot be queried, so the full step must be certified from finite stored reserves and learned snapshots; Calibration S is what makes this one-sided finite-sample certificate accept λ=1 at every own slot. Its ε3/2 critic precision is sufficient for our proof, and we do not claim it is necessary, as stated after Theorem 3. (b) Part (b) does not follow from Assumption 4. On the conflicting instance, every common stationary policy has gap at least Lconf on some family, so separate policies are necessary. The same algorithm with common supplied budgets uses one module on the compatible instance and two on the conflicting one, without testing compatibility, whereas independent policies use two on both. Theorem 3 thus gives a sufficient condition for a single deployed policy and a construction in which separate policies are necessary, and the algorithm attains the minimum module count on both instances of that construction.