acceptodds
Under review as a conference paper at ICLR 2027

How Much of a Policy Library Must Be Kept in Continual Offline RL?

Abstract

Continual offline reinforcement learning trains a specialist policy for each arriving task from static datasets, and a natural way to avoid forgetting is to keep every specialist in a growing library. Yet tasks share behavior, so how much of this library must be retained to keep serving every task seen so far? Task labels and parameter isolation cannot answer this question, since neither reveals whether an existing policy can actually solve a task. This paper answers this question with behavioral evidence. Paired-reset validation tests each candidate policy directly on each task identity and builds a directed compatibility graph. From this graph, we select a cover that decides both which policies to keep and which task each kept policy serves. The same behavioral evidence also bounds its own cost. Once every completion of the unobserved edges yields the same retention decision, further validation cannot change the outcome, and we can skip it. On 3D navigation benchmarks, our framework covers every task identity while retaining a minority of the specialists, improves service success over per-identity retention, and removes a substantial fraction of candidate validation episodes without altering any retention decision. It thus provides a simple, verifiable answer to what must be kept, and to when further validation is no longer necessary.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Offline Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CzyaOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies continual offline RL, where one specialist policy is trained per arriving task, and asks how many of these specialists must be kept so that every task seen so far can still be served. The authors use “paired-reset validation,” which runs a candidate policy and a reference policy from the same start/goal configurations, to build a directed compatibility graph whose edges mean “policy i can serve task u.” The retained library is chosen as a cover of this graph, either under a fixed budget (VTC-2) or as a minimum-cardinality adaptive cover (VBC). A query-efficient variant (QVBC) stops validating once pessimistic and optimistic completions of the unknown edges give the same retention decision. On the theory side, the paper gives a union-bound/Clopper–Pearson guarantee that every accepted edge loses at most εsaf\varepsilon_{saf} relative to the reference and meets a competence floor (Theorem 1), and a decision-equivalence result showing that early stopping reproduces full-verification decisions (Proposition 2). Experiments on SimpleTown and AmazeVille (Continual NavBench) and PointMaze (OGBench), each with 5–8 task identities, show that the adaptive cover keeps fewer modules than per-identity retention (e.g., 3 of 8 on AmazeVille) with comparable or slightly better success depending on the environment. Selective querying reduces candidate validation episodes by roughly 8–34% without changing any retention decision.

Critical strengths
  1. The question is clear and practical, and the stopping rule is a clean idea. Deciding which trained policies to keep using direct behavioral evidence, rather than parameter similarity or task labels, is easy to interpret and verify. The pessimistic/optimistic stopping rule gives exact decision preservation, and this holds on every tested prefix across all three backends.
  2. The empirical reporting is careful and transparent. The authors separate storage, training, and validation costs. They report coverage separately from success and include per-seed and per-root tables. They also openly state negative findings, for example that Fixed-4 beats the adaptive cover on AmazeVille and that the stronger 10 pp certificate is not attained.
  3. The paper explains why the library sometimes has to grow, not just that it does. The authors split extra capacity into two sources: edges removed by certification, and edges lost because earlier deletions shrink the candidate pool. On SimpleTown, this accounting (Fig. 2c) attributes almost all extra capacity to certification (28 module-prefixes, against 1 from online retention). They also build a controlled MDP with a known number of compatibility classes (Fig. 2b, Fig. 10), where the adaptive cover matches the required capacity exactly while a fixed two-slot budget loses coverage. This matters because it identifies certification strictness, not the online retention procedure, as the main cost of the certified variant.
Critical weaknesses
  1. The guarantees and the headline results do not line up, and the guarantee itself is weak. The certified acceptance rule behind Theorem 1 is used only on SimpleTown. There, certified VBC has the lowest success among the compact methods: 95.77%, against 97.27% for VTC-2 and 97.52% for per-identity retention. The strongest results, on AmazeVille and PointMaze, rely on the empirical rule, which has no guarantee. Even where the certificate applies, it is loose. The tolerance εsaf=0.20\varepsilon_{saf}=0.20 allows up to 20 percentage points of loss relative to the reference, and Figure 5 shows the worst accepted edge has a lower confidence bound near −18 pp. Both theoretical results are also fairly direct: Theorem 1 is a union bound over Clopper–Pearson intervals, and Proposition 2 is a sandwich argument. Taken together, the theory does little to support the paper’s main empirical claims.
  2. The practical benefit is small and partly explained by factors other than compression. The experiments cover only 5–8 identities, all in navigation, and the absolute storage savings are about 2–3.5 MiB. Under the validation-retention accounting (Fig. 4c, Table 23), all reference modules must be kept to continue validating future arrivals, which brings storage back to the per-identity level of 6.25 MiB. This undercuts the paper’s framing of “how much must be kept.” Validation also adds a cost the baseline never pays: per-identity retention needs zero validation episodes while VBC needs thousands, so the 8–34% query savings only reduce overhead the method itself introduces. Finally, the AmazeVille gain over per-identity retention (+4.38 pp) seems to come from some specialists being weak on their own tasks, so routing them to a better donor helps. That is a benefit of validation-based routing rather than compression, and a plain Fixed-4 budget outperforms VBC there, as the paper acknowledges.

Presentation Issues:

  • Undefined names. Baselines and abbreviations are only named and defined in Appendix C: PI-first, PI-best-validated, PI-latest, HiSPO, TSN-Core, Scratch-N, and Replay-fixed128. They appear in Table 2 without explanation. Full method descriptions can stay in the appendix, but the names and a one-line description should be in the main text. VTC-2’s “empirical guard” is also never specified.
  • Algorithm 1. It is very high level and adds little beyond the surrounding text. It should either be made concrete, for example by spelling out the QVBC query loop, or be removed or moved to the appendix.
  • Spacing artifacts. Missing paragraph spacing makes sentences run together. This happens in Lines 144-145, 362-363, 429-430.
  • Figure 3b. It is hard to read. The two numbers per bar are small and crowded, and the stacked candidate/reference encoding is not obvious.
  • Inconsistent naming. The decision-equivalence result is called “Proposition 2” in Section 3.3 but “Theorem 2” in Table 1 and Appendix B.
  • Vague abstract. It uses phrases like “a minority of the specialists” and “a substantial fraction” instead of reporting the actual numbers.
Questions for authors
  1. How does VBC compare with a non-compressing baseline that keeps all specialists but routes each identity to its best-validated policy? This would separate the benefit of validated routing from the benefit of compression. Maybe the authors can run some ablation on this.
  2. Can you report AmazeVille and PointMaze results under the certified rule? Without them, the guarantee only covers the SimpleTown setting.
  3. How do the costs of validation and of the exact cover scale as the number of identities U grows well beyond 8?
6 comments

Thank you for the careful and constructive review. Your three questions were very concrete, so we ran new experiments for each of them. We summarize the new evidence first and then answer each point. The revised PDF has been uploaded. All changes are marked in magenta.

New evidence in this response

  • Certified rule on AmazeVille, PointMaze and Meta-World (W1, Q2).
  • Keep-all validated routing, which separates the routing gain from the compression cost (W2, Q1).
  • Larger scale. AmazeVille now uses 32 training roots × 2 arrival orders × 2 validation banks (128 streams, 2048 prefixes) for capacity and query cost, and 16 roots for fresh quality. We add two Meta-World manipulation streams and a synthetic study with UU up to 128 (W2, Q3).
  • Measured selection cost and a validation-capable storage account (W2).

W1 and Q2. The guarantee and the headline results

The certified rule applies unchanged to AmazeVille, PointMaze and Meta-World. It keeps full coverage on every tested stream, and on AmazeVille it is more successful than the empirical rule at both endpoints. We use exactly the rule of Section 3.2 (α=0.10\alpha = 0.10, ε=0.20\varepsilon = 0.20, cflo=0.80c_{\text{flo}} = 0.80, n2=100n_2 = 100, n1=50n_1 = 50 reject-only stage), with the family size Emax⁡=T(2U−1)E_{\max} = T(2U-1) declared from each stream length.

Table R1. AmazeVille, 16 training roots × 2 orders × 2 validation banks. Fresh success on 500 episodes per stream and target, with root-level bootstrap 95% intervals for the gap to PI-latest.

MethodKK (P8 / P16)Success P8Success P16Gap to PI-latest, P16
PI-latest8 / 886.1886.020
VBC, empirical2.92 / 2.6691.1590.88+4.86 [3.66, 6.03]
VBC, certified4.19 / 4.1391.8692.16+6.14 [5.24, 6.92]

P8 is the endpoint after every identity has arrived once, and P16 is the endpoint after every identity has arrived twice. Certification retains about 1.3 more modules at P8 and 1.5 more at P16 than the empirical rule, and on AmazeVille this extra capacity is rewarded with higher quality. Among the 492 certified non-self routes deployed on AmazeVille, none has a fresh success below 0.80 or a fresh mean gap below −20 pp. These are point estimates on independent fresh data, so we present them as a consistency check of Theorem 1, not as a new simultaneous certificate.

On the other backends, the certified rule also keeps full coverage. On PointMaze it gives the same capacity and the same fresh success as the empirical rule (K=4K = 4 of 5, 72.3%). On the two Meta-World streams it retains 9.33 and 8.67 of 10 modules, with 91.6% and 92.7% success. Here an episode counts as a success if the task is solved at any step of the 500-step episode. SimpleTown is reported in Table 2. The certified edge guarantee is therefore applied to a deployed configuration on all four backends, not only SimpleTown.

On the 20 pp tolerance. At the default values, the binding condition is the competence floor. Because p−≤1−qp_- \leq 1 - q, a certified floor q≥0.80q \geq 0.80 already implies Δ≥−0.20\Delta \geq -0.20. For every accepted non-self edge, the certified competence floor therefore implies the default relative bound. The tolerance is a user parameter, and Theorem 1 holds for any ε\varepsilon. On AmazeVille, tightening ε\varepsilon to 0.10 at n2=100n_2 = 100 keeps full coverage on all 128 streams and raises the mean KK from 4.22 to 6.88 at P8. Stricter guarantees thus buy capacity in a predictable way, which is the trade-off the theory is meant to expose.

On the simplicity of the theory. We agree that the proof tools are standard. The contribution lies in formulating continual retention so that these tools give exact statements about the decision itself. First, prefix-local pools bound the number of potential non-self edges by T(2U−1)T(2U-1), so a finite family can be declared before the stream starts, although the tested edges are chosen adaptively. Second, the stopping rule preserves the complete decision object (donors, routes and the next candidate pool). This needs item-wise optimistic scores and an outcome-independent tie-break. Equal cover sizes alone are not enough: in Appendix Table 6, exact and greedy selection retain the same number of modules on every SimpleTown prefix, yet their routes differ on 17 of 25 prefixes of one stream. Induction then extends equality to the entire retention path. Every one of the 2048 AmazeVille prefixes, 480 Meta-World rule-prefix checks and 19,840 synthetic prefixes preserves the full decision.

W2. Practical benefit

Scale and domains. The AmazeVille study now covers 32 roots for capacity. VBC covers all identities on 128 of 128 streams with mean K=3.18K = 3.18 at P8 and K=2.86K = 2.86 at P16, and recency-first querying saves 29.2% of all validation episodes, references included, while matching full verification at all 2048 prefixes. We also add two predeclared Meta-World manipulation streams with U=10U = 10 and T=20T = 20. One repeats the standard CW10 task order, and the other contains five task pairs that differ only by an obstacle wall. Over 3 roots, VBC keeps full coverage with K=9.17K = 9.17 and 8.338.33 of 10 identities, and its success stays within 1.5 pp of per-identity retention. These manipulation tasks share little behavior, so the method keeps close to one module per task. Compression appears where behavior is shared and the library stays near UU where it is not, which is the intended behavior of an evidence-driven rule.

Absolute storage. Persistent bytes scale with K and not with the number of arrivals. On AmazeVille, the inference package shrinks from 5.93 MiB for per-identity retention to 2.59 MiB for VBC (Fig. 4c). With larger policies, the same reduction in module count gives proportionally larger absolute savings.

Validation-retention storage (Fig. 4c, Table 23). This account was overly conservative, and keeping the reference modules is not needed to continue validation. Each first-visit reference is evaluated exactly once, when its identity first arrives. Every later edge on that identity reuses the same reset manifest (allowed by Assumption A2), so validation only needs the reference's 100 binary outcomes. We replayed all 128 AmazeVille streams using only these cached outcome records, with no reference module loaded after its first evaluation. All 2048 prefixes reproduce the original donors, routes and next pools. These records take about 0.18 MiB. A validation-capable library therefore needs 2.77 MiB in total, against 6.12 MiB for per-identity retention. Fig. 4c now shows this account, and the note of Table 24 (former Table 23) states it.

Validation cost. The selection step itself is negligible: exact cover takes 2.7 ms per 16-arrival AmazeVille stream and the full recency-first query loop takes 8.3 ms. Validation is the real cost. A 16-arrival stream needs about 12.4k episodes under full verification and 8.8k under recency-first querying, compared with 11.1 wall-clock hours of specialist training per stream. At the simulator speed measured in our new runs (about 1.2 to 1.7 s per episode per process), recency-first validation is roughly a third of the training time, and it buys verified coverage of every identity.

Q3. Scaling with U

Per arrival, the potential non-self edges are at most (Ut−1)+Kt−1≤2U−1(U_t - 1) + K_{t-1} \leq 2U - 1, so validation grows linearly in UU per arrival. The exact cover runs on the local pool Lt−1∪{St}L_{t-1} \cup \{S_t\} of size Kt−1+1K_{t-1} + 1, so enumeration is exponential only in the retained capacity KK. For pools above 12 modules we solve the same lexicographic objective by MILP. We also ran 200 synthetic streams with planted compatibility classes, calibrated to accepted AmazeVille edges, for U∈{8,16,32,64,128}U \in \{8, 16, 32, 64, 128\} and T=2UT = 2U.

Table R2. Synthetic scaling, 20 instances per row.

UUClassesFinal KKCandidate episodes saved by QVBCMedian cover time per stream, full / QVBC (s)
1644.053.4%0.005 / 0.06
641615.7547.2%1.5 / 79
1283230.641.4%5.4 / 635

QVBC preserved every decision on all 19,840 prefixes. The query loop solves more covers as UU grows. However, at U=128U = 128 its extra computation is below 0.001 s per saved episode, while each simulated AmazeVille episode costs about 1.2 to 1.7 s. With the measured AmazeVille simulator rate as the cost of an episode, the validation saved is far more expensive than the extra selection computation.

Reviewer_3igyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

The authors propose a method for retaining a compact library of specialists to serve the tasks encountered in continual offline RL. They assess each specialist's ability to serve a task through behavioral validation against competence and relative-performance criteria. With these evaluations, they define a directed compatibility graph from specialist to tasks, and select a cover that determines both the retained specialists and their task assignments. Experiments on navigation benchmarks demonstrate that fewer specialists can cover all observed task identities, with improved service success on AmazeVille and modest performance losses on SimpleTown and PointMaze.

Critical strengths
  1. The paper addresses the relevant problem of policy library compression in continual offline RL. By evaluating cross-task compatibility, the proposed method reduces the number of retained specialists while maintaining task performance. Experiments across three navigation environments from two benchmarks support its effectiveness across different settings.

  2. The method is supported by theoretical guarantees, including a statistical guarantee for accepted non-self compatibility edges under stated assumptions and a decision equivalence guarantee for selective validation relative to full validation.

Critical weaknesses

While the paper studies an interesting and relevant research question, the paper’s presentation lacks sufficient detail and clarity in its description of the algorithms and definitions of key concepts, making the proposed method difficult to understand.

  1. Important algorithmic details are underspecified. The paper describes that the VTC-2 uses an empirical guard, but its update and replacement rules are unclear. Similarly, VBC is also only described at a high level. The authors do not specify how the scores are computed, or how the tie-break works. The author should either include references, or describe the specific design in detail. Appendix C briefly describes the baselines, but does not provide enough detail on baseline-specific hyperparameters
  2. Other clarity issues regarding writing: to name a few, the αloo\alpha_{loo} is not formally described as a part of the method, which can be confusing. The query schedules (recency-first, first-witness, and uniform) appear before they are formally explained. Several metrics also need clearer definitions, such as "coverage", "covered-target success", or "service success".

Moreover, the empirical evaluation does not sufficiently establish the method’s advantages, given the limited baseline comparisons, unquantified computational costs, and insufficient justification and analysis of the validation thresholds.

  1. The baseline comparisons are relatively limited. Only table 2 includes a broad set of continual-learning baselines, while studies on PointMaze and AmazeVille only compares against PI-latest or fixed-budget controls.
  2. The end-to-end computational cost of the selectors is not established. The proposed method requires environment rollouts and repeated cover selection, while per-identity retention has negligible selection overhead. Yet both of them have comparable performance in table 2. It would be important to understand the cost behind the storage savings. Although the paper reports validation episode counts, it does not really quantify selector runtime.
  3. Lastly, the validation thresholds lack sufficient justification or analysis. While the certified rule has a statistical guarantee, the final selection of the numbers are not motivated, neither does the paper investigate how these choices affect the performance. Moreover, some conditions seem to be redundant. Specifically, since p−<1−qp_- < 1- q, q≥0.8q \ge 0.8 may already imply p−≤0.2p_- \le 0.2. The authors should make the criterion more principled.
Questions for authors
  1. Could the authors provide more algorithmic details on their proposed components, such as VTC-2 and VBC?
  2. What are the validation time, cover-selection time, and total selection time of the proposed method, compared to the selected baselines?
  3. What motivates the chosen thresholds for edge compatiblity validation, and how do they affect storage savings and task success rates?
4 comments

Thank you for the careful review and for recognizing the relevance of the problem and the guarantees. We agree that the method needed a more precise specification. Below we give the full rule set, new baselines, a measured cost breakdown and a threshold sensitivity study. The revised PDF has been uploaded. All changes are marked in magenta.

W1 and Q1. Algorithmic details

Validated edge. For a candidate i and a target identity u, the candidate and the first-visit specialist of u (the reference) both run on the same 100 reset configurations. The empirical rule accepts the edge if the candidate success is at least 0.80, the paired failure rate is at most 0.20 and the mean success gap is at least −10 pp. The certified rule (Section 3.2) instead requires a Clopper-Pearson upper bound on the paired failure rate of at most 0.20 and a lower bound on candidate success of at least 0.80, each at level αloo=α/(2Emax⁡)\alpha_{\text{loo}} = \alpha / (2E_{\max}) with Emax⁡=T(2U−1)E_{\max} = T(2U-1). The −10 pp gap filter stays empirical and can only reject an edge.

Score and tie-break (VBC). An accepted edge receives the score s(i,u)=−(number of paired failures)s(i,u) = -(\text{number of paired failures}), where episodes are matched by their reset configuration. When a configuration repeats, we take the worst-case matching. Self edges need no test and have score 0. At arrival tt, VBC searches the pool Ct=Lt−1∪{St}C_t = L_{t-1} \cup \{S_t\} with a fixed lexicographic order:

  1. minimize the number of donors that cover every seen identity,
  2. among those covers, maximize the worst route score, min⁡umax⁡i∈Ds(i,u)\min_u \max_{i \in D} s(i,u),
  3. break remaining ties by the smallest donor tuple in arrival order.

Each identity is then routed to the retained donor with the highest score, and ties go to the earlier arrival. Unselected modules are deleted and never re-enter later pools.

VTC-2 guard and replacement. VTC-2 keeps at most two modules. Each new edge is validated in two stages. After 50 paired resets the edge is accepted if the mean gap is at least −5 pp and rejected if it is below −15 pp. Otherwise 50 more resets are run, and the edge is accepted only if the 100-reset gap is at least −10 pp. The edge score is the gap used for the decision. At each arrival, VTC-2 validates the retained modules on the new identity and the new specialist on every earlier identity, reusing all earlier decisions. It then keeps the feasible subset of at most two modules with the fewest modules, the best worst-case score, the best summed score and the earliest arrival order, in this priority. Replacement is therefore implicit: a retained module is dropped as soon as a better two-slot cover exists. If no two-slot cover exists, VTC-2 keeps its previous library and routes, stops, and reports that no two-slot cover exists. On the SimpleTown benchmark stream this never happened.

QVBC query loop. The revised Algorithm 1 spells out the loop:

known ← self edges of C_t (score 0) ∪ previously revealed edges
repeat
    D⁻, ρ⁻ ← ExactCover(G⁻)      # unknown non-self edges absent
    D⁺, ρ⁺ ← ExactCover(G⁺)      # unknown edges accepted with optimistic score 0
    if (D⁻, ρ⁻) = (D⁺, ρ⁺): commit and stop
    W ← unrevealed edges used by ρ⁺   # witnesses, never empty
    query one edge of W by the schedule and add its outcome to known

Schedules. Recency-first queries the witness whose target identity first appeared most recently in the stream. First-witness queries the witness with the smallest (target, donor) index. Uniform draws a legal witness at random. All schedules stop on the same condition, so they reach the same decision and differ only in cost.

W2. Definitions

  • Coverage is the fraction of seen identities that have a route.
  • Covered-target success is the mean fresh success over the routed identities only.
  • Service success is the mean fresh success over all seen identities, with every unrouted identity counted as 0. It is the single quality number for fixed budgets that abstain.

αloo\alpha_{\text{loo}}: In the submission, αloo\alpha_{\text{loo}} first appears inside Theorem 1. It is the error level of each one-sided Clopper-Pearson interval of the certified rule, that is, of each statistical look at an edge. Emax⁡=T(2U−1)E_{\max} = T(2U-1) bounds the number of potential non-self edges, and each edge uses two intervals, one for p−p_- and one for qq, so αloo=α/(2Emax⁡)\alpha_{\text{loo}} = \alpha / (2E_{\max}) keeps the family-wise error at α\alpha. The revised Section 3.2 defines it under the clearer name αlook\alpha_{\text{look}} as part of the acceptance rule, before the theorem. Section 3 also defines the reference policy, the margin filter and the route score, and the query schedules are defined before their first use. A table of baseline-specific hyperparameters is added to Appendix C (Table 8).

W3. Baselines

We report two completed groups of comparisons, and a third is in progress.

Single-network continual learners on AmazeVille. Finetune trains one high-level module through the eight arrivals. Replay adds a buffer of up to 128 episodes per past identity. Multi-task trains one module jointly on all eight datasets, which is a non-continual reference that keeps all data. All three share the frozen low-level policy and initialization of our specialists, receive no task label, and use the same total number of high-level updates (1.6M) as the specialist stream.

Table R1. AmazeVille, first 4 training roots, one arrival order, P8, 2 validation banks × 8 targets × 500 fresh episodes. Brackets give descriptive root-level bootstrap 95% intervals.

MethodRetained high-level modulesSuccess (%)Difference to VBC (pp)
VBC, empirical2.7591.930
PI-latest887.01−4.92
Finetune184.88−7.05 [−9.29, −4.17]
Replay, 128 episodes per identity179.62−12.31 [−13.38, −11.25]
Multi-task, joint180.18−11.75 [−13.26, −10.24]

Per-identity variants and keep-all routing. At P16 every AmazeVille identity has two specialists. On 16 roots × 2 orders × 2 validation banks, PI-first reaches 86.13%, PI-best-validated 86.92% and PI-latest 86.02%, while VBC reaches 90.88% with 2.66 modules. Keeping the latest specialist of each identity and routing each identity to the best validated donor among them reaches 92.80% with 8 modules. VBC thus captures most of the routing benefit with about a third of the modules. The decomposition of this gain into routing and compression is in our response to Reviewer Czya. The revised appendix adds the same per-identity variants and keep-all routing for PointMaze (Table 26).

Structural baselines. PNN, HiSPO and TSN-Core on the same AmazeVille roots are reported in our follow-up comment and in Table 28 of the revised PDF.

W4 and Q2. Computational cost

The table below breaks down the cost of one 16-arrival AmazeVille stream. Training time is the measured wall-clock time averaged over 32 roots. Validation time is the episode count multiplied by the simulator time measured in our new runs, about 1.2 to 1.7 s per episode for one simulator process. Several simulator processes can run in parallel.

Table R2. Cost per 16-arrival AmazeVille stream. Selection times are median CPU times. Validation hours are estimated from episode counts and the measured simulator rate.

StageVBC with QVBCVBC, full verificationPI-latest
Specialist training11.1 h11.1 h11.1 h
Validation episodes8.8k12.4k0
Validation time, one process2.9 to 4.2 h4.1 to 5.9 h0
Cover selection, whole stream8.3 ms2.7 ms0
Total selection time≈ validation time≈ validation time0

Cover selection is negligible, and validation is the only real overhead. With QVBC it is roughly a third of the training time. The QVBC entry includes every repeated exact solve inside the query loop. In exchange, validation verifies coverage of every identity and, on AmazeVille, raises quality at P16 by 4.9 pp over PI-latest with 33% of the modules. On a synthetic study with up to 128 identities, the extra selection computation of QVBC stays below 0.001 s per validation episode it saves.

W5 and Q3. Thresholds

Motivation. The default values were fixed before the reported evaluations and were not tuned on them. The floor cflo=0.80c_{\text{flo}} = 0.80 is an absolute competence requirement. It prevents a weak reference from licensing a weak donor, which a relative condition alone would allow. The −10 pp margin is a non-inferiority margin that tolerates a small loss relative to the reference and rejects clearly worse donors. The tolerance ε=0.20\varepsilon = 0.20 equals 1−cflo1 - c_{\text{flo}}, and α=0.10\alpha = 0.10 is the family-wise error level. With n=100n = 100 resets the certified rule is strict: on a 16-arrival AmazeVille stream, an edge needs at least 94 successes in 100 resets to certify the 0.80 floor.

Sensitivity. We varied the empirical floor and margin on AmazeVille. KK is the mean over 16 roots × 4 streams at P8.

Table R3. Empirical-rule sensitivity on AmazeVille, P8.

FloorMarginKKSuccess (%)Gap to PI-latest (pp)
0.705 pp3.1392.75+6.57
0.8010 pp2.9291.15+4.96
0.905 pp3.9492.06+5.87
0.9020 pp3.9191.92+5.74
0.7020 pp1.6985.48−0.71

The default lies in a stable region. Settings with a stricter floor or a narrower margin keep between roughly three and four modules and the full quality gain. Only the loosest combination, a low floor together with a wide margin, over-compresses the library and loses the gain. On PointMaze all four settings give the same library (K = 4, 72.3%). On SimpleTown the three other settings keep two modules on every stream, and the loosest one again drops to a single module on two of the three streams, losing 3.4 to 3.8 pp relative to the default. For the certified rule, tightening ε\varepsilon from 0.20 to 0.10 raises the mean KK from 4.2 to 6.9 on AmazeVille while coverage stays complete, so stronger guarantees buy capacity in a predictable way.

W6. Redundant conditions

You are right that q≥0.8q \geq 0.8 already implies p−≤0.2p_- \leq 0.2, because p−≤1−qp_- \leq 1 - q, and then also Δ=p+−p−≥−0.2\Delta = p_+ - p_- \geq -0.2. At the default values, the safety condition p−≤ε=0.2p_- \leq \varepsilon = 0.2 therefore never rejects an edge on its own. It becomes binding only when ε<1−cflo\varepsilon < 1 - c_{\text{flo}}. For example, under the certified rule with cflo=0.70c_{\text{flo}} = 0.70 and ε=0.20\varepsilon = 0.20, 644 edge decisions on AmazeVille are rejected by the safety condition alone. The revision states the rule in this principled form: an absolute competence floor, a reference-relative loss bound that adds information only when ε<1−cflo\varepsilon < 1 - c_{\text{flo}}, and the mean-gap margin filter as a separate reject-only check outside the guarantee.

We hope these additions address your concerns, and we would be glad to clarify anything further.

AuthorsOpenReview ↗

Follow-up on W3: structural baselines on AmazeVille

As promised, we report the structural continual-learning baselines on AmazeVille. PNN, HiSPO and TSN-Core are trained on the same four training roots, the same data and the same number of updates per task (200k) as the specialists, each with its native architecture and update rule. Evaluation uses the same fresh episodes as Table R1 of our first response. The results are in Appendix K (Table 28) of the revised PDF.

Table R4. AmazeVille, first 4 training roots, one arrival order, P8, 2 validation banks × 8 targets × 500 fresh episodes. State is the serialized model needed to act, without optimizer state. Brackets give descriptive root-level bootstrap 95% intervals.

MethodState (MiB)Success (%)Difference to VBC (pp)
VBC, empirical2.6691.930
PI-latest6.1287.01−4.92
PNN40.3192.70+0.78 [−0.36, +1.91]
HiSPO10.1688.00−3.93 [−5.68, −2.30]
TSN-Core7.6934.31−57.62 [−64.72, −50.83]
Finetune1.3684.88−7.05 [−9.29, −4.17]
Replay1.3679.62−12.31 [−13.38, −11.25]
Multi-task1.3680.18−11.75 [−13.26, −10.24]

VBC offers the best trade-off between success and stored state. Only PNN reaches the success of VBC. The interval of the difference contains zero, and PNN stores about 15 times as much state because it adds a full column for every task. HiSPO stores about 4 times as much state and is 3.9 pp below VBC. TSN-Core does not retain the tasks in this setting. It was also the weakest structural baseline on SimpleTown (Table 2). The single-network learners store the least and are 7 to 12 pp below VBC.

Together with the per-identity variants and keep-all routing in our first response, AmazeVille now has the same breadth of baselines as Table 2. The PointMaze comparisons are in Table 26 of the revised appendix.

Reviewer_1uebOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates the problem of compact policy retention in continual offline reinforcement learning, aiming to preserve performance on previously encountered tasks while reducing the size of the policy library. The authors propose a behavioral validation framework that uses paired-reset evaluations to construct a directed compatibility graph, based on which a subset of policies is selected to cover historical tasks. Experiments on three navigation environments demonstrate that the proposed approach can substantially reduce the number of retained policies and validation episodes while maintaining task coverage and competitive performance.

Critical strengths
  1. The motivation of this paper is interesting, as some tasks share common behaviors, making it unnecessary to retain a separate policy for every task.
  2. The figures and tables are well-designed and visually appealing, providing a clear and intuitive presentation of the proposed method and experimental results.
Critical weaknesses
  1. I find the problem formulation somewhat puzzling. Why does continual RL require maintaining a separate specialized policy for each task, rather than continually training a single model to acquire and retain the capabilities needed for all tasks? The latter seems to be a more natural and widely accepted setting for continual learning.
  2. The experimental settings are relatively simple and limited to navigation tasks, which raises concerns about the generalizability of the proposed method.
  3. There is a spacing issue between lines 429 and 430, and the numbers in Figure 3(b) are excessively large.
  4. The writing and presentation of the paper could be significantly improved in terms of clarity and readability. In particular, the Introduction contains many unnecessarily complex sentences with excessive subordinate clauses, making it difficult to follow the main arguments.
Questions for authors

See Weaknesses.

3 comments

Thank you for your review and for finding the motivation interesting. Your main questions concern the problem formulation and the scope of the experiments, and we answer both with new experiments. The certified rule on all backends, keep-all routing baselines and measured costs are in our responses to Reviewers 3igy and Czya. The revised PDF has been uploaded. All changes are marked in magenta.

W1. Why keep a library of policies instead of one network?

Both settings are established in continual RL, and they make different trade-offs. One line trains a single network and protects old skills by replay or regularization. Another line stores task-specific components and selects them by task identity, for example progressive networks (PNN), PackNet, Expert Gate, policy subspaces (CSP, HiSPO) and TSN. In offline continual RL each task arrives with a fixed dataset, so a specialist can be trained for each task without further interaction. Keeping the specialists never overwrites an old skill, but the library grows with every task. Our paper asks how much of such a library must actually be kept. The question applies to any continual learner that keeps a library of specialist policies. Our rule also returns a single module whenever one policy in the candidate pool covers every seen identity.

On AmazeVille, single-network learners are clearly less successful. We trained three single-network learners on AmazeVille, which has eight distinct identities that each arrive once. Finetune trains one high-level module through the eight tasks. Replay adds a buffer of up to 128 episodes per past task. Multi-task trains one module jointly on all eight datasets and keeps all data. All three use the same frozen low-level policy and initialization as our specialists, receive no task label, and use the same total number of high-level updates (1.6M) as the specialist stream.

Table R1. AmazeVille, first 4 training roots, one arrival order, after all 8 identities have arrived. 2 validation banks × 8 targets × 500 fresh episodes. Brackets give descriptive root-level bootstrap 95% intervals.

MethodRetained high-level modulesSuccess (%)Difference to VBC (pp)
VBC (ours)2.7591.930
Per-identity, latest887.01−4.92
Finetune184.88−7.05 [−9.29, −4.17]
Replay179.62−12.31 [−13.38, −11.25]
Multi-task180.18−11.75 [−13.26, −10.24]

With the same data and update budget, VBC is 7 to 12 pp more successful than every single-network learner while keeping fewer than three modules. Finetune also forgets. On 100 fixed episodes per identity, earlier identities lose 4.4 pp on average between their own arrival and the end of the stream. Replay and joint training keep past data, yet they do not close the gap.

When tasks share behavior, both approaches work well. On the SimpleTown stream (Table 2 of the paper), Finetune reaches 97.79% with 13.35 MiB of persistent state, and VTC-2 reaches 97.27% with 2.03 MiB. Here validation shows that two policies serve all identities, so the library stays small. Verified reuse removes a specialist only when validation shows that the retained policies still serve its task.

W2. Beyond navigation

We added two manipulation streams from Meta-World. Both have 10 identities and 20 arrivals. The first repeats the standard Continual World task order (CW10) twice. The second contains five pairs of tasks that differ only by an obstacle wall. Specialists are MLP policies trained by behavior cloning from noisy scripted demonstrations. This tests a manipulation simulator and flat MLP policies, while the edge rule, thresholds and query rule are unchanged.

Table R2. Meta-World at the end of each stream, 3 training roots × 2 validation banks, 200 fresh episodes per target. Each cell gives K / success (%). An episode counts as a success if the task is solved at any step of the 500-step episode.

StreamVBC, empiricalVBC, certifiedPer-identity, latest
CW10 order, twice9.17 / 91.459.33 / 91.5810 / 92.20
Wall pairs8.33 / 92.498.67 / 92.7310 / 93.92

The method transfers without any change. It covers every identity on all streams, and the query-efficient variant reproduces the full-verification decision at all 480 rule-prefix checks. These manipulation tasks need distinct motions, so validation finds little shared behavior. VBC then keeps close to one module per identity, with success within 1.5 pp of per-identity retention. This is the intended behavior: the library shrinks only where validation shows that behavior is shared, as on AmazeVille and SimpleTown.

Further scope.

  • PointMaze (Section 4.4) already uses a different simulator (MuJoCo from OGBench) and a different learner (goal-conditioned implicit Q-learning).
  • AmazeVille now uses 32 training roots × 2 arrival orders × 2 validation banks, which gives 128 streams. This includes a 16-arrival setting where every identity arrives twice. VBC covers every identity on all 128 streams with 3.18 modules after 8 arrivals and 2.86 after 16.
  • A synthetic study scales the number of identities up to 128. The query-efficient rule preserves every decision on all 19,840 prefixes.

W3. Formatting

Thank you. The revision fixes the spacing between lines 429 and 430, together with similar issues at lines 144-145 and 362-363, and replaces Figure 3 with a single readable panel.

W4. Clarity of the Introduction

The revised Introduction uses short sentences and states the problem directly. Its new opening reads:

A continual learner receives tasks one after another. In offline RL, each task comes with a fixed dataset, so the learner can train one specialist policy per task. Keeping every specialist protects old skills, but the library grows with every task. Many tasks can be solved by the same behavior, so many specialists are redundant. We ask how many policies must be kept so that every task seen so far is still solved well.

To answer this, we test behavior directly. A stored policy may serve another task only if it succeeds on that task almost as often as the task's own specialist, measured from the same start states. These tests define a graph. A small cover of this graph gives both the retained library and the policy that serves each task. Testing costs environment interaction, so we also show when testing can stop early without changing the decision.

The revision applies the same style to the rest of the paper, describes every baseline in one line in the main text, and states the main numbers in the abstract.

We hope these results clarify the formulation and the scope of the method, and we would be glad to answer any further questions.

AuthorsOpenReview ↗

Follow-up on W1: single-network learners on Meta-World

Thank you again for your question about training a single network. We have now completed the same comparison on the two Meta-World manipulation streams, so the question is answered outside navigation as well.

Finetune trains one network through the 20 arrivals. Multi-task trains one network jointly on all data, so it is a non-continual reference that must keep every dataset. Both use the same network, optimizer, data and total number of updates as the specialists, and receive no task label.

Table R3. Meta-World at the end of each stream, 3 training roots × 2 validation banks × 10 targets × 200 fresh episodes. Success means that the task is solved at any step of the 500-step episode.

MethodModulesCW10 order, twiceWall pairs
VBC, empirical9.17 / 8.3391.4592.49
VBC, certified9.33 / 8.6791.5892.73
Finetune16.643.48
Multi-task, keeps all data195.0175.62

A single network trained sequentially does not retain these tasks. Finetune ends at 6.6% and 3.5%. Measured on the same episodes, its success on a task falls by 31 pp and 52 pp on average between the first time the task is trained and the end of the stream.

Joint training is not a reliable substitute. It needs all past data at once. It is 3.6 pp above VBC on the first stream and 16.9 pp below VBC on the wall stream, where paired tasks need different motions in similar states.

VBC is the only method in this table that is above 91% on both streams without storing past data. Together with the AmazeVille result in our first response, this supports the library formulation: keeping validated specialists gives stable quality, and the cover decides how many of them are needed. These results are in Appendix K (Table 27) of the revised PDF.

Comments on the paper

1 comment
AuthorsOpenReview ↗

A single model and a library are complementary

Thank you for your reply. We are glad that the experiments address your concerns, and we answer the remaining question on the problem formulation.

We agree that training one model through several stages is the dominant approach, and our formulation builds on it rather than competing with it. Even large single models are often extended with small task-specific modules, for example libraries of LoRA adapters on one pretrained model (LoraHub, Huang et al., 2023, and Ostapenko et al., 2024, "Towards Modular LLMs by Building and Reusing a Library of LoRAs"). Our AmazeVille specialists have the same structure. All of them share one low-level policy, and each task adds only a small high-level module. The question we study is which of these task-specific modules must be kept, and VBC answers it with behavioral evidence.

A single model is also a valid candidate for VBC, and we have now tested this directly. The rule accepts any policy that passes validation on a task. If one model serves every seen task, VBC keeps only that model, so the single-model solution corresponds to a library of size one. If the model has lost some earlier tasks, validation reveals this, and VBC keeps an earlier snapshot of the model only for those tasks.

We applied VBC, unchanged, to the snapshots of the Finetune network on AmazeVille. After each arrival, the current snapshot of the single network is the new candidate, and each snapshot is validated on the tasks it has already seen. Selection uses 100 validation episodes per task, and success is measured on a separate set of fresh episodes.

Table R8. AmazeVille, first 4 training roots, one arrival order, after all 8 identities have arrived. 8 targets × 500 fresh episodes from an evaluation bank not used for selection. This is one of the two banks of Table R1, so the numbers differ slightly from Table R1. The bracket gives a descriptive root-level bootstrap 95% interval.

MethodRetained high-level modulesSuccess (%)
Finetune, final model only185.09
VBC over Finetune snapshots3.0092.23 (+7.14 [+4.39, +9.89] over Finetune)
VBC over specialists2.7592.44
Per-identity, latest886.97

On every root, the final model fails validation on some earlier tasks, and VBC keeps one to three earlier snapshots for them. This recovers 7.1 pp of success and comes close to VBC over specialists. Validation therefore tells when one model is enough, and when it is not, it keeps exactly the snapshots that are still needed. This makes VBC a complement to single-model continual learning rather than an alternative to it.

In offline continual RL, one model is often not enough. Each task arrives as a fixed dataset, and the earlier data may no longer be available. With the same data and number of updates, sequential training of one network ends below 7% success on both Meta-World streams, whereas the library stays above 91% (Table 27). On AmazeVille, every single-network learner is 7 to 12 pp below VBC (Table 28).

The updated PDF adds this discussion to the Introduction and the snapshot experiment to Appendix K.9 (Table 32). Thank you again for your feedback, which helped us state the motivation more clearly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.