- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies continual offline RL, where one specialist policy is trained per arriving task, and asks how many of these specialists must be kept so that every task seen so far can still be served. The authors use “paired-reset validation,” which runs a candidate policy and a reference policy from the same start/goal configurations, to build a directed compatibility graph whose edges mean “policy i can serve task u.” The retained library is chosen as a cover of this graph, either under a fixed budget (VTC-2) or as a minimum-cardinality adaptive cover (VBC). A query-efficient variant (QVBC) stops validating once pessimistic and optimistic completions of the unknown edges give the same retention decision. On the theory side, the paper gives a union-bound/Clopper–Pearson guarantee that every accepted edge loses at most relative to the reference and meets a competence floor (Theorem 1), and a decision-equivalence result showing that early stopping reproduces full-verification decisions (Proposition 2). Experiments on SimpleTown and AmazeVille (Continual NavBench) and PointMaze (OGBench), each with 5–8 task identities, show that the adaptive cover keeps fewer modules than per-identity retention (e.g., 3 of 8 on AmazeVille) with comparable or slightly better success depending on the environment. Selective querying reduces candidate validation episodes by roughly 8–34% without changing any retention decision.
Critical strengths
- The question is clear and practical, and the stopping rule is a clean idea. Deciding which trained policies to keep using direct behavioral evidence, rather than parameter similarity or task labels, is easy to interpret and verify. The pessimistic/optimistic stopping rule gives exact decision preservation, and this holds on every tested prefix across all three backends.
- The empirical reporting is careful and transparent. The authors separate storage, training, and validation costs. They report coverage separately from success and include per-seed and per-root tables. They also openly state negative findings, for example that Fixed-4 beats the adaptive cover on AmazeVille and that the stronger 10 pp certificate is not attained.
- The paper explains why the library sometimes has to grow, not just that it does. The authors split extra capacity into two sources: edges removed by certification, and edges lost because earlier deletions shrink the candidate pool. On SimpleTown, this accounting (Fig. 2c) attributes almost all extra capacity to certification (28 module-prefixes, against 1 from online retention). They also build a controlled MDP with a known number of compatibility classes (Fig. 2b, Fig. 10), where the adaptive cover matches the required capacity exactly while a fixed two-slot budget loses coverage. This matters because it identifies certification strictness, not the online retention procedure, as the main cost of the certified variant.
Critical weaknesses
- The guarantees and the headline results do not line up, and the guarantee itself is weak. The certified acceptance rule behind Theorem 1 is used only on SimpleTown. There, certified VBC has the lowest success among the compact methods: 95.77%, against 97.27% for VTC-2 and 97.52% for per-identity retention. The strongest results, on AmazeVille and PointMaze, rely on the empirical rule, which has no guarantee. Even where the certificate applies, it is loose. The tolerance allows up to 20 percentage points of loss relative to the reference, and Figure 5 shows the worst accepted edge has a lower confidence bound near −18 pp. Both theoretical results are also fairly direct: Theorem 1 is a union bound over Clopper–Pearson intervals, and Proposition 2 is a sandwich argument. Taken together, the theory does little to support the paper’s main empirical claims.
- The practical benefit is small and partly explained by factors other than compression. The experiments cover only 5–8 identities, all in navigation, and the absolute storage savings are about 2–3.5 MiB. Under the validation-retention accounting (Fig. 4c, Table 23), all reference modules must be kept to continue validating future arrivals, which brings storage back to the per-identity level of 6.25 MiB. This undercuts the paper’s framing of “how much must be kept.” Validation also adds a cost the baseline never pays: per-identity retention needs zero validation episodes while VBC needs thousands, so the 8–34% query savings only reduce overhead the method itself introduces. Finally, the AmazeVille gain over per-identity retention (+4.38 pp) seems to come from some specialists being weak on their own tasks, so routing them to a better donor helps. That is a benefit of validation-based routing rather than compression, and a plain Fixed-4 budget outperforms VBC there, as the paper acknowledges.
Presentation Issues:
- Undefined names. Baselines and abbreviations are only named and defined in Appendix C: PI-first, PI-best-validated, PI-latest, HiSPO, TSN-Core, Scratch-N, and Replay-fixed128. They appear in Table 2 without explanation. Full method descriptions can stay in the appendix, but the names and a one-line description should be in the main text. VTC-2’s “empirical guard” is also never specified.
- Algorithm 1. It is very high level and adds little beyond the surrounding text. It should either be made concrete, for example by spelling out the QVBC query loop, or be removed or moved to the appendix.
- Spacing artifacts. Missing paragraph spacing makes sentences run together. This happens in Lines 144-145, 362-363, 429-430.
- Figure 3b. It is hard to read. The two numbers per bar are small and crowded, and the stacked candidate/reference encoding is not obvious.
- Inconsistent naming. The decision-equivalence result is called “Proposition 2” in Section 3.3 but “Theorem 2” in Table 1 and Appendix B.
- Vague abstract. It uses phrases like “a minority of the specialists” and “a substantial fraction” instead of reporting the actual numbers.
Questions for authors
- How does VBC compare with a non-compressing baseline that keeps all specialists but routes each identity to its best-validated policy? This would separate the benefit of validated routing from the benefit of compression. Maybe the authors can run some ablation on this.
- Can you report AmazeVille and PointMaze results under the certified rule? Without them, the guarantee only covers the SimpleTown setting.
- How do the costs of validation and of the exact cover scale as the number of identities U grows well beyond 8?
6 comments
On the 20 pp tolerance. At the default values, the binding condition is the competence floor. Because , a certified floor already implies . For every accepted non-self edge, the certified competence floor therefore implies the default relative bound. The tolerance is a user parameter, and Theorem 1 holds for any . On AmazeVille, tightening to 0.10 at keeps full coverage on all 128 streams and raises the mean from 4.22 to 6.88 at P8. Stricter guarantees thus buy capacity in a predictable way, which is the trade-off the theory is meant to expose.
On the simplicity of the theory. We agree that the proof tools are standard. The contribution lies in formulating continual retention so that these tools give exact statements about the decision itself. First, prefix-local pools bound the number of potential non-self edges by , so a finite family can be declared before the stream starts, although the tested edges are chosen adaptively. Second, the stopping rule preserves the complete decision object (donors, routes and the next candidate pool). This needs item-wise optimistic scores and an outcome-independent tie-break. Equal cover sizes alone are not enough: in Appendix Table 6, exact and greedy selection retain the same number of modules on every SimpleTown prefix, yet their routes differ on 17 of 25 prefixes of one stream. Induction then extends equality to the entire retention path. Every one of the 2048 AmazeVille prefixes, 480 Meta-World rule-prefix checks and 19,840 synthetic prefixes preserves the full decision.
W2. Practical benefit
Scale and domains. The AmazeVille study now covers 32 roots for capacity. VBC covers all identities on 128 of 128 streams with mean at P8 and at P16, and recency-first querying saves 29.2% of all validation episodes, references included, while matching full verification at all 2048 prefixes. We also add two predeclared Meta-World manipulation streams with and . One repeats the standard CW10 task order, and the other contains five task pairs that differ only by an obstacle wall. Over 3 roots, VBC keeps full coverage with and of 10 identities, and its success stays within 1.5 pp of per-identity retention. These manipulation tasks share little behavior, so the method keeps close to one module per task. Compression appears where behavior is shared and the library stays near where it is not, which is the intended behavior of an evidence-driven rule.
Absolute storage. Persistent bytes scale with K and not with the number of arrivals. On AmazeVille, the inference package shrinks from 5.93 MiB for per-identity retention to 2.59 MiB for VBC (Fig. 4c). With larger policies, the same reduction in module count gives proportionally larger absolute savings.
Validation-retention storage (Fig. 4c, Table 23). This account was overly conservative, and keeping the reference modules is not needed to continue validation. Each first-visit reference is evaluated exactly once, when its identity first arrives. Every later edge on that identity reuses the same reset manifest (allowed by Assumption A2), so validation only needs the reference's 100 binary outcomes. We replayed all 128 AmazeVille streams using only these cached outcome records, with no reference module loaded after its first evaluation. All 2048 prefixes reproduce the original donors, routes and next pools. These records take about 0.18 MiB. A validation-capable library therefore needs 2.77 MiB in total, against 6.12 MiB for per-identity retention. Fig. 4c now shows this account, and the note of Table 24 (former Table 23) states it.
Validation cost. The selection step itself is negligible: exact cover takes 2.7 ms per 16-arrival AmazeVille stream and the full recency-first query loop takes 8.3 ms. Validation is the real cost. A 16-arrival stream needs about 12.4k episodes under full verification and 8.8k under recency-first querying, compared with 11.1 wall-clock hours of specialist training per stream. At the simulator speed measured in our new runs (about 1.2 to 1.7 s per episode per process), recency-first validation is roughly a third of the training time, and it buys verified coverage of every identity.
Q3. Scaling with U
Per arrival, the potential non-self edges are at most , so validation grows linearly in per arrival. The exact cover runs on the local pool of size , so enumeration is exponential only in the retained capacity . For pools above 12 modules we solve the same lexicographic objective by MILP. We also ran 200 synthetic streams with planted compatibility classes, calibrated to accepted AmazeVille edges, for and .
Table R2. Synthetic scaling, 20 instances per row.
| Classes | Final | Candidate episodes saved by QVBC | Median cover time per stream, full / QVBC (s) | |
|---|---|---|---|---|
| 16 | 4 | 4.0 | 53.4% | 0.005 / 0.06 |
| 64 | 16 | 15.75 | 47.2% | 1.5 / 79 |
| 128 | 32 | 30.6 | 41.4% | 5.4 / 635 |
QVBC preserved every decision on all 19,840 prefixes. The query loop solves more covers as grows. However, at its extra computation is below 0.001 s per saved episode, while each simulated AmazeVille episode costs about 1.2 to 1.7 s. With the measured AmazeVille simulator rate as the cost of an episode, the validation saved is far more expensive than the extra selection computation.
Thank you for the careful and constructive review. Your three questions were very concrete, so we ran new experiments for each of them. We summarize the new evidence first and then answer each point. The revised PDF has been uploaded. All changes are marked in magenta.
New evidence in this response
W1 and Q2. The guarantee and the headline results
The certified rule applies unchanged to AmazeVille, PointMaze and Meta-World. It keeps full coverage on every tested stream, and on AmazeVille it is more successful than the empirical rule at both endpoints. We use exactly the rule of Section 3.2 (α=0.10, ε=0.20, cflo=0.80, n2=100, n1=50 reject-only stage), with the family size Emax=T(2U−1) declared from each stream length.
Table R1. AmazeVille, 16 training roots × 2 orders × 2 validation banks. Fresh success on 500 episodes per stream and target, with root-level bootstrap 95% intervals for the gap to PI-latest.
P8 is the endpoint after every identity has arrived once, and P16 is the endpoint after every identity has arrived twice. Certification retains about 1.3 more modules at P8 and 1.5 more at P16 than the empirical rule, and on AmazeVille this extra capacity is rewarded with higher quality. Among the 492 certified non-self routes deployed on AmazeVille, none has a fresh success below 0.80 or a fresh mean gap below −20 pp. These are point estimates on independent fresh data, so we present them as a consistency check of Theorem 1, not as a new simultaneous certificate.
On the other backends, the certified rule also keeps full coverage. On PointMaze it gives the same capacity and the same fresh success as the empirical rule (K=4 of 5, 72.3%). On the two Meta-World streams it retains 9.33 and 8.67 of 10 modules, with 91.6% and 92.7% success. Here an episode counts as a success if the task is solved at any step of the 500-step episode. SimpleTown is reported in Table 2. The certified edge guarantee is therefore applied to a deployed configuration on all four backends, not only SimpleTown.