- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies model-based control under recurring dynamics, distinguishing first visits to new dynamics from revisits to previously encountered dynamics. The proposed method, FSSL, retains a frozen pretrained model alongside a learning copy that is continually updated through multiple-choice learning over trajectory segments. Prediction heads are selected for planning based on recent prediction error. Experiments cover six environments, with FSSL achieving the highest mean performance in the final evaluation with frozen weights after the deployment stream. The paper also provides analyses by phase and conditional bounds on planning error to examine the roles of continual learning and preservation of the pretrained model.
Critical strengths
The analysis by phase is valuable. The paper shows that continual learning can improve performance on revisits while degrading performance when first entering new dynamics. This helps identify adaptation costs that aggregate metrics can obscure and provides a useful perspective for evaluating learning during deployment.
The method has empirical support. FSSL achieves the highest mean performance in the final evaluation across all six environments. Component ablations support the value of slow learning in some environments, and the frozen copy improves performance on first-visit entries in HalfCheetah. The authors also clearly report nonsignificant results and methodological limitations.
Critical weaknesses
The technical advance over prior work is limited. Multiple dynamics heads, multiple-choice learning over trajectory segments, and head selection based on recent prediction error are already present in T-MCL. The main additions are persistent updates and a frozen copy. Table 3 further shows that removing Gaussian variance prediction from T-MCL substantially narrows the gap to the frozen FSSL checkpoint. Although the component ablations support some design choices, the contribution of the proposed combination relative to strengthened versions of existing methods remains insufficiently established.
The effectiveness of the actual selection mechanism has not been directly validated. Figure 4 identifies which copy contains the lowest-error head on a fixed set of state-action pairs, rather than reporting the planner’s actual selections during control. The theoretical results also rely on conditions such as selecting an appropriate prediction head. These results therefore do not sufficiently establish that the intended division of roles across phases occurs during actual control. Moreover, the full method has not demonstrated a clear advantage over its frozen checkpoint across the entire deployment stream in the three analyzed environments, limiting the evidence for the practical value of the added mechanism.
Questions for authors
Mechanism validation: Could the authors report the proportions of frozen and learning heads actually selected during planning, separately for first-visit entries, the remaining episodes of cycle one, and revisits? Are these actual selections consistent with the division of roles observed on the fixed evaluation sets in Figure 4?
Attribution of the gains: For the T-MCL variant without Gaussian variance prediction in Table 3, could the authors provide paired comparisons with FSSL on first-visit entries, revisits, and the entire deployment stream? Together with the existing ablations using the same checkpoint, could these comparisons further distinguish the gains from simplifying the pretrained model from those attributable to persistent learning and the frozen copy?
4 comments
Response to Reviewer BWmp (2/4): Question 2
Reply to Question 2: Attribution against T-MCL without Gaussian variance
Thank you for this question. Table R4 gives the paired comparisons you asked for, between FSSL and T-MCL without the Gaussian variance, kept frozen and given our update. Intersection, SlimHumanoid and Hopper were added during the discussion.
FSSL against the variant. The reward gain comes from our stream update, not from a margin of FSSL over the variant. Against the frozen variant, FSSL is higher on Intersection, by +0.322 [+0.239, +0.405] on revisits and +0.293 [+0.196, +0.383] over the whole stream. It is lower on CartPole, by 0.021 at entries and 0.006 on revisits. The entry, revisit and whole-stream intervals on the other four streams include zero, with a positive mean on Ant and negative means on the other three. Once the variant receives our update, it is level with FSSL or above it on every stream, by 0.009 [0.005, 0.013] on CartPole and 0.988 [0.410, 1.494] on HalfCheetah over the whole stream. All six final differences from the updated variant include zero, with the higher mean on its side.
The same update on the variant. Our update raises the whole-stream reward of the frozen variant on three streams, by +0.672 [+0.511, +0.833] on HalfCheetah, +0.306 [+0.197, +0.426] on Intersection and +0.050 [+0.014, +0.087] on Ant. Both sides share the simplified pretrained model, so these gains come with our stream update and not with the simplification. By phase, the update changes HalfCheetah entry and revisit reward by −0.668 [−1.044, −0.121] and +0.939 [+0.655, +1.220], the asymmetry of our claim. Intersection shows the revisit gain alone, +0.350 [+0.259, +0.450], while the phase intervals on the other four streams include zero. Their revisit means are positive on Ant, SlimHumanoid and Hopper, with entry means of mixed sign.
What the comparison supports and our revision. With the Gaussian variance removed on both sides, the update still adds reward, which separates the simplification from the update, while the frozen copy is separated on the FSSL checkpoint alone (Appendix Table D.3). Our main claim is the asymmetry of learning on the stream, which is in Theorem 1 and the phase analysis. The revision removes the sentence of Section 4.2 that credited the remaining gap to slow learning and the frozen copy (lines 500–511). The phase analysis of Figures 3 and 4 now opens Section 4.2 ahead of Table 1 (lines 377–458), with the comparison against the variant placed after the component results. The comparisons are in lines 1104–1112 and Appendix Table D.6 (page 22).
Table R4. T-MCL without the Gaussian variance on the deployment stream, per-step differences over matched episodes of five seeds, with 90% seed-cluster bootstrap intervals. "With update" gives the variant our stream update with the buffer, budget, selection and planner of FSSL w/o frozen copy. The table condenses Appendix Table D.6 of the revised paper, which adds the final evaluation and the rest of cycle one.
| Comparison | Stream | Entries | Revisits | Whole stream |
|---|---|---|---|---|
| FSSL − variant, frozen | CartPole | −0.021 [−0.036, −0.006] | −0.006 [−0.010, −0.004] | −0.008 [−0.012, −0.005] |
| HalfCheetah | −0.356 [−1.038, +0.326] | −0.054 [−0.837, +0.681] | −0.316 [−0.952, +0.320] | |
| Ant | +0.036 [−0.072, +0.144] | +0.022 [−0.005, +0.048] | +0.032 [−0.015, +0.076] | |
| Intersection | −0.074 [−0.264, +0.113] | +0.322 [+0.239, +0.405] | +0.293 [+0.196, +0.383] | |
| SlimHumanoid | −0.938 [−2.398, +0.599] | −0.539 [−1.460, +0.335] | −0.523 [−1.139, +0.053] | |
| Hopper | −0.047 [−0.256, +0.112] | −0.021 [−0.151, +0.108] | −0.024 [−0.127, +0.079] | |
| FSSL − variant, with update | CartPole | −0.014 [−0.021, −0.008] | −0.006 [−0.012, −0.001] | −0.009 [−0.013, −0.005] |
| HalfCheetah | +0.312 [−0.530, +1.249] | −0.992 [−1.596, −0.388] | −0.988 [−1.494, −0.410] | |
| Ant | −0.037 [−0.159, +0.092] | −0.013 [−0.044, +0.019] | −0.019 [−0.039, +0.005] | |
| Intersection | −0.099 [−0.266, +0.069] | −0.027 [−0.109, +0.054] | −0.012 [−0.104, +0.073] | |
| SlimHumanoid | −1.849 [−3.320, −0.505] | −0.837 [−2.479, +0.970] | −0.724 [−2.033, +0.509] | |
| Hopper | +0.113 [−0.057, +0.318] | −0.094 [−0.301, +0.108] | −0.058 [−0.205, +0.110] | |
| Variant with update − variant, frozen | CartPole | −0.006 [−0.017, +0.004] | +0.000 [−0.003, +0.004] | +0.000 [−0.003, +0.003] |
| HalfCheetah | −0.668 [−1.044, −0.121] | +0.939 [+0.655, +1.220] | +0.672 [+0.511, +0.833] | |
| Ant | +0.072 [−0.070, +0.215] | +0.035 [−0.000, +0.074] | +0.050 [+0.014, +0.087] | |
| Intersection | +0.025 [−0.155, +0.205] | +0.350 [+0.259, +0.450] | +0.306 [+0.197, +0.426] | |
| SlimHumanoid | +0.911 [−1.449, +3.551] | +0.299 [−0.956, +1.600] | +0.201 [−1.014, +1.464] | |
| Hopper | −0.160 [−0.560, +0.148] | +0.073 [−0.011, +0.157] | +0.034 [−0.037, +0.105] |
Response to Reviewer BWmp (3/4): Weakness 2a and Question 1
Reply to Weakness 2a and Question 1: Validation of the selection mechanism
Thank you for raising this. Table R2 gives the proportions you asked for, the share of frozen heads among the heads that the planner rolls out, in the three phases. The shares come from the head that each member rolls out, logged at every stream step of all thirty seeds (lines 430–458, with Figure 4 on page 8 and Appendix Table D.7 on page 22).
Results. Members select frozen heads more at first-visit entries than on revisits during actual control. Over thirty seeds the frozen share is 0.591 to 0.975 at entries, against 0.090 to 0.506 on revisits. The entry share is higher in each of thirty seeds on every stream.
Table R2. Share of frozen heads among the heads that the planner rolls out during control. The share is averaged over steps and members within an episode, then over the episodes of a phase and over seeds 0 to 29. The last row counts the seeds with a higher share at first-visit entries than on revisits.
| Phase | CartPole | HalfCheetah | Ant | Intersection | SlimHumanoid | Hopper |
|---|---|---|---|---|---|---|
| First-visit entries | 0.793 | 0.975 | 0.765 | 0.835 | 0.664 | 0.591 |
| Rest of cycle one | 0.245 | 0.363 | 0.674 | 0.224 | 0.557 | 0.436 |
| Revisits | 0.275 | 0.267 | 0.506 | 0.090 | 0.322 | 0.288 |
| Seeds, entries > revisits | 30/30 | 30/30 | 30/30 | 30/30 | 30/30 | 30/30 |
Consistency with Figure 4. The actual selections agree in direction with the fixed-set shares of Figure 4 of the submission. Over the six streams the planner gap is narrower on five (Appendix Table D.7). On HalfCheetah the planner share is 0.975 at entries and 0.267 on revisits, against fixed-set shares of 1.000 and 0.133. The division is clear on CartPole, HalfCheetah and Intersection, whereas on Ant and SlimHumanoid the mean frozen share stays above one half in the rest of cycle one. On CartPole, HalfCheetah and Intersection one learning head of FSSL w/o frozen copy holds 0.93, 0.70 and 0.89 of the credit of a mode, which meets the premise of the revisit bound in Theorem 1 (i) only in part.
Learning heads on revisits. Over five seeds, a member returns on revisits to one learning head on four of the six streams, mostly the head of lowest fixed-set error. Both patterns are weak on Ant and Hopper (Appendix Table D.10, page 25).
Response to Reviewer BWmp (4/4): Weakness 2b
Reply to Weakness 2b: Whole stream against the frozen checkpoint
We thank the reviewer for this point. Table R7 extends the comparison of FSSL with its own frozen checkpoint from five seeds on three streams to thirty seeds on six. FSSL now exceeds its frozen checkpoint over the whole stream on CartPole and Intersection, by +0.006 [+0.003, +0.008] and +0.238 [+0.197, +0.280]. The other four means are +0.055 on HalfCheetah, +0.001 on Ant, +0.329 on SlimHumanoid and −0.024 on Hopper, with intervals that include zero.
Why the whole-stream mean is small. Relative to its frozen checkpoint, FSSL raises reward on revisits and in the final evaluation on CartPole, HalfCheetah and Intersection, which the whole-stream mean dilutes with cycle one. The whole-stream mean averages first visits with revisits, the two phases that the paper analyses apart. Cycle one fills one third of the stream on five streams and one sixth on Intersection. On HalfCheetah the differences are −0.121 at entries, −0.329 in the rest of cycle one and +0.229 on revisits, which gives +0.055 [−0.065, +0.179] over the stream. After the stream the final-score difference is +0.579 [+0.417, +0.744], positive in 26 of 30 seeds. The contribution that we claim is the asymmetry between the two phases, in which slow learning gains on revisits at a first-visit cost that the frozen copy reduces.
The frozen copy at first visits. Without the frozen copy, learning loses 0.905 at HalfCheetah entries and 0.267 at Intersection entries against the frozen checkpoint (Table R1). With the frozen copy these differences are −0.121 and −0.080, with intervals that include zero. The direct comparison at entries gives +0.784 [+0.478, +1.097] and +0.186 [+0.110, +0.261] in favour of the frozen copy (last column of Table R7). Over the whole stream the same comparison gives +0.040 [+0.016, +0.064] on Hopper, while the other five intervals include zero (Appendix Table D.3). The frozen copy thus reduces the entry losses on HalfCheetah and Intersection at no measurable cost over the stream.
Limits. FSSL still loses to its checkpoint at first-visit entries on Ant and Hopper, by 0.064 [0.015, 0.114] and 0.085 [0.042, 0.127]. On both streams the credit of a mode is spread over the learning heads of FSSL w/o frozen copy, with a top-head share of 0.41 and 0.45 against one third for an even split. This spread matches the missing revisit gain on Ant and the revisit loss on Hopper without the frozen copy (Table R1). Frozen heads receive 0.765 and 0.591 of the entry selections there, so learning heads still enter the plan (Table R2). This points to a selection rule that holds frozen heads at first visits as the next step. These results are included in the revised paper (lines 460–467 and Appendix Table D.3, page 20).
Table R7. FSSL minus its own frozen checkpoint by phase, per-step reward over matched episodes of seeds 0 to 29, with 90% seed-cluster bootstrap intervals. Parentheses give the number of seeds that share the sign of the mean final difference. The last column gives FSSL minus FSSL w/o frozen copy at first-visit entries.
| Stream | Entries | Rest of cycle one | Revisits | Whole stream | Final | Entries, FSSL − FSSL w/o frozen copy |
|---|---|---|---|---|---|---|
| CartPole | −0.003 [−0.009, +0.003] | +0.005 [+0.002, +0.009] | +0.007 [+0.004, +0.010] | +0.006 [+0.003, +0.008] | +0.010 [+0.007, +0.013] (28/30) | +0.001 [−0.005, +0.006] |
| HalfCheetah | −0.121 [−0.360, +0.124] | −0.329 [−0.517, −0.140] | +0.229 [+0.082, +0.379] | +0.055 [−0.065, +0.179] | +0.579 [+0.417, +0.744] (26/30) | +0.784 [+0.478, +1.097] |
| Ant | −0.064 [−0.114, −0.015] | −0.008 [−0.032, +0.016] | +0.013 [−0.005, +0.030] | +0.001 [−0.013, +0.014] | +0.026 [−0.007, +0.059] (18/30) | +0.025 [−0.027, +0.080] |
| Intersection | −0.080 [−0.172, +0.015] | +0.074 [+0.013, +0.137] | +0.278 [+0.234, +0.321] | +0.238 [+0.197, +0.280] | +0.338 [+0.276, +0.398] (29/30) | +0.186 [+0.110, +0.261] |
| SlimHumanoid | +0.759 [−0.082, +1.599] | +0.055 [−0.576, +0.702] | +0.378 [−0.081, +0.855] | +0.329 [−0.090, +0.761] | +0.273 [−0.304, +0.867] (13/30) | +0.611 [−0.184, +1.413] |
| Hopper | −0.085 [−0.127, −0.042] | −0.033 [−0.073, +0.005] | −0.013 [−0.049, +0.023] | −0.024 [−0.054, +0.005] | −0.017 [−0.053, +0.020] (18/30) | −0.007 [−0.058, +0.042] |
We hope that our response has addressed your concerns. As the discussion period closes soon, we will reply promptly to any further question.
Response to Reviewer BWmp (1/4): Weakness 1
Dear Reviewer BWmp,
Thank you for your careful review and for recognising our phase analysis and experiments. In the revised paper, we extend the phase analysis to thirty seeds (W1) and report paired comparisons against the T-MCL variant (Q2). We also report the planner selections during control (W2a, Q1) and the whole-stream comparison with the frozen checkpoint (W2b). Letters mark the parts of one paragraph of your review, in its order. We have addressed your concerns as follows.
Reply to Weakness 1: Technical advance over T-MCL
We thank the reviewer for this point. Although we acknowledge that multiple heads, segment credit and the selection rule come from T-MCL, our advance still lies in a controller that keeps learning on the deployment stream and limits the reward lost at first visits. With weights fixed at deployment, T-MCL serves a returning mode with the same inference as a first visit. FSSL w/o frozen copy gains on revisits on three streams but loses at first-visit entries on four (Table R1), a loss consistent with predictions that have moved away from pretraining.
With the frozen copy, the revisit gain stays on CartPole, HalfCheetah and Intersection, with a smaller entry loss on HalfCheetah and Intersection. Behind it is our central claim, that learning on the stream can lower reward at first-visit entries while raising it on revisits, relative to the same checkpoint kept frozen (Figure 3). We support this claim in analysis and in experiments. Theorem 1 and Corollary 1.1 bound the planning gap of each phase, with Theorem 2 in the revised paper supplementing the corollary for the selection rule.
Evidence. Our entry and revisit analysis now covers thirty seeds. Seven of the twelve differences are significant under the paired t-test with the sign of the claim, which extends the pattern of the submission on HalfCheetah (Table R1). A T-MCL variant given the same update shows the same pattern on HalfCheetah, losing 0.668 at entries and gaining 0.939 on revisits over five seeds (Table R4). The planner in turn selects frozen heads more often at first-visit entries than on revisits in each of thirty seeds on all six streams (Appendix Table D.7). The revised paper presents these results at the start of Section 4.2 (lines 377–458), with the abstract and the introduction revised to match.
Table R1. FSSL w/o frozen copy minus its own frozen checkpoint, per-step reward over matched episodes of seeds 0 to 29. Brackets give the 90% seed-cluster bootstrap interval. p is that of the paired t-test over seed means. Bold marks p < 0.05. W is the credit separation over seeds 0 to 29, the distance between the credit shares of two modes over the learning heads, from 0 for one split to 1 for disjoint heads. The table condenses Appendix Tables D.3 and D.4 of the revised paper.