acceptodds
Under review as a conference paper at ICLR 2027

First Visits and Revisits: Learning from the Deployment Stream in Model-Based Control

Abstract

A controller facing recurring dynamics meets first visits to new dynamics and revisits to dynamics that its stream has shown. We argue that adaptation then runs on two time scales, fast selection among hypotheses the model holds and slow learning of those hypotheses, which compete for the same weights. We propose FSSL (Fast Selection, Slow Learning), a model-based controller whose members keep a frozen copy of their pretrained prediction heads beside a copy that one multiple-choice loss trains between episodes. Each member plans with the head of lowest recent error among both copies. On six streams against eight baselines, FSSL gives the best mean on all six, and slow learning, hard credit and per-head weighting each raise its score significantly. Without the frozen copy, the learning heads lose on first-visit entries and gain on revisits of CartPole and HalfCheetah, whereas with it a frozen head best explains the entered mode for 79 to 100% of the members. Planning-error bounds per phase explain this division of labour. Code and demo are available at anonymous.4open.science/r/fssl.

Primary area:
applications to robotics, autonomy, planning
Keywords:
model-based reinforcement learning, non-stationary dynamics, recurring dynamics, continual adaptation, multiple-choice learning, fast and slow adaptation, model predictive control, catastrophic forgetting

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BWmpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies model-based control under recurring dynamics, distinguishing first visits to new dynamics from revisits to previously encountered dynamics. The proposed method, FSSL, retains a frozen pretrained model alongside a learning copy that is continually updated through multiple-choice learning over trajectory segments. Prediction heads are selected for planning based on recent prediction error. Experiments cover six environments, with FSSL achieving the highest mean performance in the final evaluation with frozen weights after the deployment stream. The paper also provides analyses by phase and conditional bounds on planning error to examine the roles of continual learning and preservation of the pretrained model.

Critical strengths

The analysis by phase is valuable. The paper shows that continual learning can improve performance on revisits while degrading performance when first entering new dynamics. This helps identify adaptation costs that aggregate metrics can obscure and provides a useful perspective for evaluating learning during deployment.

The method has empirical support. FSSL achieves the highest mean performance in the final evaluation across all six environments. Component ablations support the value of slow learning in some environments, and the frozen copy improves performance on first-visit entries in HalfCheetah. The authors also clearly report nonsignificant results and methodological limitations.

Critical weaknesses

The technical advance over prior work is limited. Multiple dynamics heads, multiple-choice learning over trajectory segments, and head selection based on recent prediction error are already present in T-MCL. The main additions are persistent updates and a frozen copy. Table 3 further shows that removing Gaussian variance prediction from T-MCL substantially narrows the gap to the frozen FSSL checkpoint. Although the component ablations support some design choices, the contribution of the proposed combination relative to strengthened versions of existing methods remains insufficiently established.

The effectiveness of the actual selection mechanism has not been directly validated. Figure 4 identifies which copy contains the lowest-error head on a fixed set of state-action pairs, rather than reporting the planner’s actual selections during control. The theoretical results also rely on conditions such as selecting an appropriate prediction head. These results therefore do not sufficiently establish that the intended division of roles across phases occurs during actual control. Moreover, the full method has not demonstrated a clear advantage over its frozen checkpoint across the entire deployment stream in the three analyzed environments, limiting the evidence for the practical value of the added mechanism.

Questions for authors

Mechanism validation: Could the authors report the proportions of frozen and learning heads actually selected during planning, separately for first-visit entries, the remaining episodes of cycle one, and revisits? Are these actual selections consistent with the division of roles observed on the fixed evaluation sets in Figure 4?

Attribution of the gains: For the T-MCL variant without Gaussian variance prediction in Table 3, could the authors provide paired comparisons with FSSL on first-visit entries, revisits, and the entire deployment stream? Together with the existing ablations using the same checkpoint, could these comparisons further distinguish the gains from simplifying the pretrained model from those attributable to persistent learning and the frozen copy?

4 comments
AuthorsOpenReview ↗

Response to Reviewer BWmp (1/4): Weakness 1

Dear Reviewer BWmp,

Thank you for your careful review and for recognising our phase analysis and experiments. In the revised paper, we extend the phase analysis to thirty seeds (W1) and report paired comparisons against the T-MCL variant (Q2). We also report the planner selections during control (W2a, Q1) and the whole-stream comparison with the frozen checkpoint (W2b). Letters mark the parts of one paragraph of your review, in its order. We have addressed your concerns as follows.

Reply to Weakness 1: Technical advance over T-MCL

We thank the reviewer for this point. Although we acknowledge that multiple heads, segment credit and the selection rule come from T-MCL, our advance still lies in a controller that keeps learning on the deployment stream and limits the reward lost at first visits. With weights fixed at deployment, T-MCL serves a returning mode with the same inference as a first visit. FSSL w/o frozen copy gains on revisits on three streams but loses at first-visit entries on four (Table R1), a loss consistent with predictions that have moved away from pretraining.

With the frozen copy, the revisit gain stays on CartPole, HalfCheetah and Intersection, with a smaller entry loss on HalfCheetah and Intersection. Behind it is our central claim, that learning on the stream can lower reward at first-visit entries while raising it on revisits, relative to the same checkpoint kept frozen (Figure 3). We support this claim in analysis and in experiments. Theorem 1 and Corollary 1.1 bound the planning gap of each phase, with Theorem 2 in the revised paper supplementing the corollary for the selection rule.

Evidence. Our entry and revisit analysis now covers thirty seeds. Seven of the twelve differences are significant under the paired t-test with the sign of the claim, which extends the pattern of the submission on HalfCheetah (Table R1). A T-MCL variant given the same update shows the same pattern on HalfCheetah, losing 0.668 at entries and gaining 0.939 on revisits over five seeds (Table R4). The planner in turn selects frozen heads more often at first-visit entries than on revisits in each of thirty seeds on all six streams (Appendix Table D.7). The revised paper presents these results at the start of Section 4.2 (lines 377–458), with the abstract and the introduction revised to match.

Table R1. FSSL w/o frozen copy minus its own frozen checkpoint, per-step reward over matched episodes of seeds 0 to 29. Brackets give the 90% seed-cluster bootstrap interval. p is that of the paired t-test over seed means. Bold marks p < 0.05. W is the credit separation over seeds 0 to 29, the distance between the credit shares of two modes over the learning heads, from 0 for one split to 1 for disjoint heads. The table condenses Appendix Tables D.3 and D.4 of the revised paper.

StreamEntriespRevisitspW
CartPole−0.003 [−0.009, +0.002]0.342+0.009 [+0.007, +0.012]<0.0010.78
HalfCheetah−0.905 [−1.256, −0.550]<0.001+0.333 [+0.209, +0.455]<0.0010.73
Ant−0.089 [−0.145, −0.031]0.018−0.004 [−0.030, +0.023]0.8180.10
Intersection−0.267 [−0.366, −0.164]<0.001+0.283 [+0.232, +0.332]<0.0010.58
SlimHumanoid+0.148 [−0.729, +1.027]0.789+0.412 [−0.187, +1.010]0.2770.15
Hopper−0.078 [−0.131, −0.024]0.027−0.065 [−0.103, −0.028]0.0100.18
AuthorsOpenReview ↗

Response to Reviewer BWmp (2/4): Question 2

Reply to Question 2: Attribution against T-MCL without Gaussian variance

Thank you for this question. Table R4 gives the paired comparisons you asked for, between FSSL and T-MCL without the Gaussian variance, kept frozen and given our update. Intersection, SlimHumanoid and Hopper were added during the discussion.

FSSL against the variant. The reward gain comes from our stream update, not from a margin of FSSL over the variant. Against the frozen variant, FSSL is higher on Intersection, by +0.322 [+0.239, +0.405] on revisits and +0.293 [+0.196, +0.383] over the whole stream. It is lower on CartPole, by 0.021 at entries and 0.006 on revisits. The entry, revisit and whole-stream intervals on the other four streams include zero, with a positive mean on Ant and negative means on the other three. Once the variant receives our update, it is level with FSSL or above it on every stream, by 0.009 [0.005, 0.013] on CartPole and 0.988 [0.410, 1.494] on HalfCheetah over the whole stream. All six final differences from the updated variant include zero, with the higher mean on its side.

The same update on the variant. Our update raises the whole-stream reward of the frozen variant on three streams, by +0.672 [+0.511, +0.833] on HalfCheetah, +0.306 [+0.197, +0.426] on Intersection and +0.050 [+0.014, +0.087] on Ant. Both sides share the simplified pretrained model, so these gains come with our stream update and not with the simplification. By phase, the update changes HalfCheetah entry and revisit reward by −0.668 [−1.044, −0.121] and +0.939 [+0.655, +1.220], the asymmetry of our claim. Intersection shows the revisit gain alone, +0.350 [+0.259, +0.450], while the phase intervals on the other four streams include zero. Their revisit means are positive on Ant, SlimHumanoid and Hopper, with entry means of mixed sign.

What the comparison supports and our revision. With the Gaussian variance removed on both sides, the update still adds reward, which separates the simplification from the update, while the frozen copy is separated on the FSSL checkpoint alone (Appendix Table D.3). Our main claim is the asymmetry of learning on the stream, which is in Theorem 1 and the phase analysis. The revision removes the sentence of Section 4.2 that credited the remaining gap to slow learning and the frozen copy (lines 500–511). The phase analysis of Figures 3 and 4 now opens Section 4.2 ahead of Table 1 (lines 377–458), with the comparison against the variant placed after the component results. The comparisons are in lines 1104–1112 and Appendix Table D.6 (page 22).

Table R4. T-MCL without the Gaussian variance on the deployment stream, per-step differences over matched episodes of five seeds, with 90% seed-cluster bootstrap intervals. "With update" gives the variant our stream update with the buffer, budget, selection and planner of FSSL w/o frozen copy. The table condenses Appendix Table D.6 of the revised paper, which adds the final evaluation and the rest of cycle one.

ComparisonStreamEntriesRevisitsWhole stream
FSSL − variant, frozenCartPole−0.021 [−0.036, −0.006]−0.006 [−0.010, −0.004]−0.008 [−0.012, −0.005]
HalfCheetah−0.356 [−1.038, +0.326]−0.054 [−0.837, +0.681]−0.316 [−0.952, +0.320]
Ant+0.036 [−0.072, +0.144]+0.022 [−0.005, +0.048]+0.032 [−0.015, +0.076]
Intersection−0.074 [−0.264, +0.113]+0.322 [+0.239, +0.405]+0.293 [+0.196, +0.383]
SlimHumanoid−0.938 [−2.398, +0.599]−0.539 [−1.460, +0.335]−0.523 [−1.139, +0.053]
Hopper−0.047 [−0.256, +0.112]−0.021 [−0.151, +0.108]−0.024 [−0.127, +0.079]
FSSL − variant, with updateCartPole−0.014 [−0.021, −0.008]−0.006 [−0.012, −0.001]−0.009 [−0.013, −0.005]
HalfCheetah+0.312 [−0.530, +1.249]−0.992 [−1.596, −0.388]−0.988 [−1.494, −0.410]
Ant−0.037 [−0.159, +0.092]−0.013 [−0.044, +0.019]−0.019 [−0.039, +0.005]
Intersection−0.099 [−0.266, +0.069]−0.027 [−0.109, +0.054]−0.012 [−0.104, +0.073]
SlimHumanoid−1.849 [−3.320, −0.505]−0.837 [−2.479, +0.970]−0.724 [−2.033, +0.509]
Hopper+0.113 [−0.057, +0.318]−0.094 [−0.301, +0.108]−0.058 [−0.205, +0.110]
Variant with update − variant, frozenCartPole−0.006 [−0.017, +0.004]+0.000 [−0.003, +0.004]+0.000 [−0.003, +0.003]
HalfCheetah−0.668 [−1.044, −0.121]+0.939 [+0.655, +1.220]+0.672 [+0.511, +0.833]
Ant+0.072 [−0.070, +0.215]+0.035 [−0.000, +0.074]+0.050 [+0.014, +0.087]
Intersection+0.025 [−0.155, +0.205]+0.350 [+0.259, +0.450]+0.306 [+0.197, +0.426]
SlimHumanoid+0.911 [−1.449, +3.551]+0.299 [−0.956, +1.600]+0.201 [−1.014, +1.464]
Hopper−0.160 [−0.560, +0.148]+0.073 [−0.011, +0.157]+0.034 [−0.037, +0.105]
AuthorsOpenReview ↗

Response to Reviewer BWmp (3/4): Weakness 2a and Question 1

Reply to Weakness 2a and Question 1: Validation of the selection mechanism

Thank you for raising this. Table R2 gives the proportions you asked for, the share of frozen heads among the heads that the planner rolls out, in the three phases. The shares come from the head that each member rolls out, logged at every stream step of all thirty seeds (lines 430–458, with Figure 4 on page 8 and Appendix Table D.7 on page 22).

Results. Members select frozen heads more at first-visit entries than on revisits during actual control. Over thirty seeds the frozen share is 0.591 to 0.975 at entries, against 0.090 to 0.506 on revisits. The entry share is higher in each of thirty seeds on every stream.

Table R2. Share of frozen heads among the heads that the planner rolls out during control. The share is averaged over steps and members within an episode, then over the episodes of a phase and over seeds 0 to 29. The last row counts the seeds with a higher share at first-visit entries than on revisits.

PhaseCartPoleHalfCheetahAntIntersectionSlimHumanoidHopper
First-visit entries0.7930.9750.7650.8350.6640.591
Rest of cycle one0.2450.3630.6740.2240.5570.436
Revisits0.2750.2670.5060.0900.3220.288
Seeds, entries > revisits30/3030/3030/3030/3030/3030/30

Consistency with Figure 4. The actual selections agree in direction with the fixed-set shares of Figure 4 of the submission. Over the six streams the planner gap is narrower on five (Appendix Table D.7). On HalfCheetah the planner share is 0.975 at entries and 0.267 on revisits, against fixed-set shares of 1.000 and 0.133. The division is clear on CartPole, HalfCheetah and Intersection, whereas on Ant and SlimHumanoid the mean frozen share stays above one half in the rest of cycle one. On CartPole, HalfCheetah and Intersection one learning head of FSSL w/o frozen copy holds 0.93, 0.70 and 0.89 of the credit of a mode, which meets the premise of the revisit bound in Theorem 1 (i) only in part.

Learning heads on revisits. Over five seeds, a member returns on revisits to one learning head on four of the six streams, mostly the head of lowest fixed-set error. Both patterns are weak on Ant and Hopper (Appendix Table D.10, page 25).

AuthorsOpenReview ↗

Response to Reviewer BWmp (4/4): Weakness 2b

Reply to Weakness 2b: Whole stream against the frozen checkpoint

We thank the reviewer for this point. Table R7 extends the comparison of FSSL with its own frozen checkpoint from five seeds on three streams to thirty seeds on six. FSSL now exceeds its frozen checkpoint over the whole stream on CartPole and Intersection, by +0.006 [+0.003, +0.008] and +0.238 [+0.197, +0.280]. The other four means are +0.055 on HalfCheetah, +0.001 on Ant, +0.329 on SlimHumanoid and −0.024 on Hopper, with intervals that include zero.

Why the whole-stream mean is small. Relative to its frozen checkpoint, FSSL raises reward on revisits and in the final evaluation on CartPole, HalfCheetah and Intersection, which the whole-stream mean dilutes with cycle one. The whole-stream mean averages first visits with revisits, the two phases that the paper analyses apart. Cycle one fills one third of the stream on five streams and one sixth on Intersection. On HalfCheetah the differences are −0.121 at entries, −0.329 in the rest of cycle one and +0.229 on revisits, which gives +0.055 [−0.065, +0.179] over the stream. After the stream the final-score difference is +0.579 [+0.417, +0.744], positive in 26 of 30 seeds. The contribution that we claim is the asymmetry between the two phases, in which slow learning gains on revisits at a first-visit cost that the frozen copy reduces.

The frozen copy at first visits. Without the frozen copy, learning loses 0.905 at HalfCheetah entries and 0.267 at Intersection entries against the frozen checkpoint (Table R1). With the frozen copy these differences are −0.121 and −0.080, with intervals that include zero. The direct comparison at entries gives +0.784 [+0.478, +1.097] and +0.186 [+0.110, +0.261] in favour of the frozen copy (last column of Table R7). Over the whole stream the same comparison gives +0.040 [+0.016, +0.064] on Hopper, while the other five intervals include zero (Appendix Table D.3). The frozen copy thus reduces the entry losses on HalfCheetah and Intersection at no measurable cost over the stream.

Limits. FSSL still loses to its checkpoint at first-visit entries on Ant and Hopper, by 0.064 [0.015, 0.114] and 0.085 [0.042, 0.127]. On both streams the credit of a mode is spread over the learning heads of FSSL w/o frozen copy, with a top-head share of 0.41 and 0.45 against one third for an even split. This spread matches the missing revisit gain on Ant and the revisit loss on Hopper without the frozen copy (Table R1). Frozen heads receive 0.765 and 0.591 of the entry selections there, so learning heads still enter the plan (Table R2). This points to a selection rule that holds frozen heads at first visits as the next step. These results are included in the revised paper (lines 460–467 and Appendix Table D.3, page 20).

Table R7. FSSL minus its own frozen checkpoint by phase, per-step reward over matched episodes of seeds 0 to 29, with 90% seed-cluster bootstrap intervals. Parentheses give the number of seeds that share the sign of the mean final difference. The last column gives FSSL minus FSSL w/o frozen copy at first-visit entries.

StreamEntriesRest of cycle oneRevisitsWhole streamFinalEntries, FSSL − FSSL w/o frozen copy
CartPole−0.003 [−0.009, +0.003]+0.005 [+0.002, +0.009]+0.007 [+0.004, +0.010]+0.006 [+0.003, +0.008]+0.010 [+0.007, +0.013] (28/30)+0.001 [−0.005, +0.006]
HalfCheetah−0.121 [−0.360, +0.124]−0.329 [−0.517, −0.140]+0.229 [+0.082, +0.379]+0.055 [−0.065, +0.179]+0.579 [+0.417, +0.744] (26/30)+0.784 [+0.478, +1.097]
Ant−0.064 [−0.114, −0.015]−0.008 [−0.032, +0.016]+0.013 [−0.005, +0.030]+0.001 [−0.013, +0.014]+0.026 [−0.007, +0.059] (18/30)+0.025 [−0.027, +0.080]
Intersection−0.080 [−0.172, +0.015]+0.074 [+0.013, +0.137]+0.278 [+0.234, +0.321]+0.238 [+0.197, +0.280]+0.338 [+0.276, +0.398] (29/30)+0.186 [+0.110, +0.261]
SlimHumanoid+0.759 [−0.082, +1.599]+0.055 [−0.576, +0.702]+0.378 [−0.081, +0.855]+0.329 [−0.090, +0.761]+0.273 [−0.304, +0.867] (13/30)+0.611 [−0.184, +1.413]
Hopper−0.085 [−0.127, −0.042]−0.033 [−0.073, +0.005]−0.013 [−0.049, +0.023]−0.024 [−0.054, +0.005]−0.017 [−0.053, +0.020] (18/30)−0.007 [−0.058, +0.042]

We hope that our response has addressed your concerns. As the discussion period closes soon, we will reply promptly to any further question.

Reviewer_8cRWOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies model-based control in the setting where the dynamics cycle among a small set of recurring modes in equal length trajectories. The authors split deployment episodes into first visits, where the current mode has not yet appeared in the stream, and revisits, where transitions from that mode are already in the replay buffer. The authors propose that first visits need fast inference over hypotheses formed during pretraining, while revisits benefit from slowly fitting those hypotheses to the buffered data. The proposed method, FSSL ("Fast Selection, Slow Learning"), uses an ensemble of dynamics networks, each with a shared trunk and three deterministic prediction heads. The heads are pretrained on source dynamics with a segment-level winner-take-all loss, encouraging head specialization resulting in a variant of T-MCL. At test-time, the controller maintains a frozen copy and a learning copy trained with gradient descent after every episode, and planning is done with the head incurring the lowest loss on recent transitions. The authors also present some theoretical results bounding the planning gap in this setting. FSSL is validated experimentally on 6 tasks based on the performance of the final controller along with ablations.

Critical strengths
  • The overall framing of the paper is good. The authors present sufficient empirical evidence to demonstrate that the problem is real and relevant. The decomposition makes precise how a stream average can hide a first-visit loss. The frozen copy is a cheap (no added training params) solution with planning cost changing only negligibly.
  • The authors include a relatively strong baseline ablation showing that removing T-MCL's Gaussian variance alone recovers most of the gap to the FSSL checkpoint which is a useful finding for the community.
Critical weaknesses
  • While the problem being studied is interesting and the proposed approach is simple, the results indicate that the central mechanism provides only marginal benefits. For instance, on Table 2, the frozen copy has no significant effect, and even in the deployment setting it stays close to the pretrained performance. All the empirical results are generally quite weak. Much of the gains of FSSL are recovered by removing Gaussian variance of T-MCL on most tasks, and another closely related approach FAMLE is only run without adaptation.
  • Narendra & Balakrishnan (1997), which the paper cites, already proposed combining fixed and adaptive models and switching to whichever has the lowest recent identification error and T-MCL already selects heads by recent error so the novelty seems limited.
  • Corollary 1.1 assumes every member selects a head of θ0\theta_0, which seems like exactly the behaviour that needs to be shown. Lemma 1.1(i) bounds the error by Nk mk(θ)=(∑igθ(xi)2)1/2\sqrt{N_k\, m_k(\theta)} = \big(\sum_i g_\theta(x_i)^2\big)^{1/2}. For a fixed mkm_k this grows with the buffer size, contradicting the intended message that more revisit data help.
Questions for authors
  • Can you report deployment-stream reward broken down by phase (first-visit entry, rest of cycle one, revisits) for all methods on all six streams and not only FSSL variants on three? Ideally also cumulative regret against an oracle with mode labels. Table 3 reports T-MCL without the Gaussian variance on only three streams. Can you add it to Table 1 for all six, alongside the frozen FSSL checkpoint?
  • Table 2 shows that slow learning adds nothing measurable on Ant, Hopper or SlimHumanoid. Why, and what does FSSL offer on those streams beyond a deterministic checkpoint?
  • The conclusion notes that measuring frozen-head shares on the planner's own selections "would test this division more directly". Can you report how often the planner actually selects a head of θ0\theta_0 in each phase, and how this compares with the fixed-set shares in Fig. 4?
6 comments
AuthorsOpenReview ↗

Response to Reviewer 8cRW (1/4): Weakness 1a

Dear Reviewer 8cRW,

Thank you for your thorough review and for your positive view of the problem framing and the low cost of the frozen copy. In the revised paper, we compare learning with its frozen checkpoint over thirty seeds (W1a), run T-MCL without the Gaussian variance on all six streams (W1b, Q1c) and report FAMLE with its fine-tuning (W1c). We clarify the novelty (W2), restate Corollary 1.1 and Lemma 1.1 (W3a, W3b), compare all eight baselines by phase (Q1a), add the oracle with mode labels (Q1b), examine the three streams without a gain (Q2) and report the planner selections (Q3). Letters mark the parts of one bullet of your review, in its order. We have addressed your concerns as follows.

Reply to Weakness 1a: Benefit of the central mechanism

We thank the reviewer for this point. We agree that the frozen copy has no significant effect on the final score in Table 2. Rather than a higher final score, our central claim is that learning on the stream can lower reward at first-visit entries while raising it on revisits, relative to the same checkpoint kept frozen (Figure 3).

Evidence. We extended the phase comparison from five seeds on three streams to thirty seeds on six (Table R1). Both effects are now clear on two streams. Learning lowers entry reward by 0.905 and raises revisit reward by 0.333 on HalfCheetah. The same two changes are 0.267 and 0.283 on Intersection, each of the four with p < 0.001 under the paired t-test. Over the six streams, seven of the twelve entry and revisit differences are significant with the sign of the claim. These results extend the pattern that the submission reported on HalfCheetah and the revisit gain on CartPole (lines 377–428).

Table R1. FSSL w/o frozen copy minus its own frozen checkpoint, per-step reward over matched episodes of seeds 0 to 29. Brackets give the 90% seed-cluster bootstrap interval. p is that of the paired t-test over seed means. Bold marks p < 0.05. W is the credit separation over seeds 0 to 29, the distance between the credit shares of two modes over the learning heads, from 0 for one split to 1 for disjoint heads. The table condenses Appendix Tables D.3 and D.4 of the revised paper.

StreamEntriespRevisitspW
CartPole−0.003 [−0.009, +0.002]0.342+0.009 [+0.007, +0.012]<0.0010.78
HalfCheetah−0.905 [−1.256, −0.550]<0.001+0.333 [+0.209, +0.455]<0.0010.73
Ant−0.089 [−0.145, −0.031]0.018−0.004 [−0.030, +0.023]0.8180.10
Intersection−0.267 [−0.366, −0.164]<0.001+0.283 [+0.232, +0.332]<0.0010.58
SlimHumanoid+0.148 [−0.729, +1.027]0.789+0.412 [−0.187, +1.010]0.2770.15
Hopper−0.078 [−0.131, −0.024]0.027−0.065 [−0.103, −0.028]0.0100.18

Components. We agree that the component ablations do not show consistent final gains across streams. Under the paired t-test, slow learning raises the final score on CartPole and HalfCheetah, while hard credit and per-head weighting do so on CartPole (Table 2, lines 483–498). The revised abstract therefore no longer presents each component as a significant gain, with Figures 3 and 4 now placed ahead of Table 1.

Frozen copy. The frozen copy acts where learning costs reward, at first-visit entries. Over thirty seeds FSSL exceeds the learning heads alone there by +0.784 [+0.478, +1.097] on HalfCheetah and +0.186 [+0.110, +0.261] on Intersection (Appendix Table D.3, lines 460–467 of the revised paper). The final evaluation measures revisits alone, which is why Table 2 shows no significant effect of the frozen copy. Over the whole stream the same comparison gives +0.040 [+0.016, +0.064] on Hopper, while the other five intervals include zero, as entries fill one twelfth of a stream or less. The frozen copy thus reduces the entry losses on HalfCheetah and Intersection at no measurable cost over the stream.

AuthorsOpenReview ↗

Response to Reviewer 8cRW (2/4): Weakness 1b and Question 1c

Reply to Weakness 1b and Question 1c: T-MCL without Gaussian variance

Thank you for raising this. As you asked (Q1c), T-MCL without the Gaussian variance now runs on all six streams, frozen and with our update (Table R4, condensed from Appendix Table D.6). The frozen FSSL checkpoint is the reference of the paired differences on the same six streams (Appendix Table D.3). We agree that this variant closes most of the gap to FSSL in final score. Rather than a margin over this variant, our central claim is that learning on the stream can lower entry reward while raising revisit reward.

Results. Against the frozen variant, FSSL is higher on Intersection, by +0.322 [+0.239, +0.405] on revisits and +0.368 [+0.268, +0.468] in the final evaluation. It is lower on CartPole, while the entry, revisit and whole-stream intervals on the other four streams include zero. Once the variant receives our update, it is level with FSSL or above it on every stream, by 0.009 on CartPole and 0.988 on HalfCheetah over the whole stream. All six final differences from the updated variant include zero, a row given in Appendix Table D.6 alone.

The same update on the variant. Our update raises the whole-stream reward of the frozen variant on three streams, by +0.672 [+0.511, +0.833] on HalfCheetah, +0.306 [+0.197, +0.426] on Intersection and +0.050 [+0.014, +0.087] on Ant. Both sides of this comparison lack the Gaussian variance, so these gains come with our stream update and not with the simplification. By phase the update lowers the entry reward of the variant and raises its revisit reward on HalfCheetah, by 0.668 and 0.939, which is the asymmetry of our claim.

Placement. Table 1 lists the eight published baselines, whereas the variant is a control built for this comparison. It therefore sits in Appendix Tables D.2 and D.6, with a summary in Section 4.2 (lines 500–511). That summary replaces the submitted sentence that credited the remaining gap to slow learning and the frozen copy.

Table R4. T-MCL without the Gaussian variance on the deployment stream, per-step differences over matched episodes of five seeds, with 90% seed-cluster bootstrap intervals. "With update" gives the variant our stream update with the buffer, budget, selection and planner of FSSL w/o frozen copy. The table condenses Appendix Table D.6 of the revised paper, which adds the final evaluation and the rest of cycle one.

ComparisonStreamEntriesRevisitsWhole stream
FSSL − variant, frozenCartPole−0.021 [−0.036, −0.006]−0.006 [−0.010, −0.004]−0.008 [−0.012, −0.005]
HalfCheetah−0.356 [−1.038, +0.326]−0.054 [−0.837, +0.681]−0.316 [−0.952, +0.320]
Ant+0.036 [−0.072, +0.144]+0.022 [−0.005, +0.048]+0.032 [−0.015, +0.076]
Intersection−0.074 [−0.264, +0.113]+0.322 [+0.239, +0.405]+0.293 [+0.196, +0.383]
SlimHumanoid−0.938 [−2.398, +0.599]−0.539 [−1.460, +0.335]−0.523 [−1.139, +0.053]
Hopper−0.047 [−0.256, +0.112]−0.021 [−0.151, +0.108]−0.024 [−0.127, +0.079]
FSSL − variant, with updateCartPole−0.014 [−0.021, −0.008]−0.006 [−0.012, −0.001]−0.009 [−0.013, −0.005]
HalfCheetah+0.312 [−0.530, +1.249]−0.992 [−1.596, −0.388]−0.988 [−1.494, −0.410]
Ant−0.037 [−0.159, +0.092]−0.013 [−0.044, +0.019]−0.019 [−0.039, +0.005]
Intersection−0.099 [−0.266, +0.069]−0.027 [−0.109, +0.054]−0.012 [−0.104, +0.073]
SlimHumanoid−1.849 [−3.320, −0.505]−0.837 [−2.479, +0.970]−0.724 [−2.033, +0.509]
Hopper+0.113 [−0.057, +0.318]−0.094 [−0.301, +0.108]−0.058 [−0.205, +0.110]
Variant with update − variant, frozenCartPole−0.006 [−0.017, +0.004]+0.000 [−0.003, +0.004]+0.000 [−0.003, +0.003]
HalfCheetah−0.668 [−1.044, −0.121]+0.939 [+0.655, +1.220]+0.672 [+0.511, +0.833]
Ant+0.072 [−0.070, +0.215]+0.035 [−0.000, +0.074]+0.050 [+0.014, +0.087]
Intersection+0.025 [−0.155, +0.205]+0.350 [+0.259, +0.450]+0.306 [+0.197, +0.426]
SlimHumanoid+0.911 [−1.449, +3.551]+0.299 [−0.956, +1.600]+0.201 [−1.014, +1.464]
Hopper−0.160 [−0.560, +0.148]+0.073 [−0.011, +0.157]+0.034 [−0.037, +0.105]
AuthorsOpenReview ↗

Response to Reviewer 8cRW (3/4): Weaknesses 1c, 2, 3a and 3b

Reply to Weakness 1c: FAMLE with adaptation

We thank the reviewer for pointing this out. FAMLE now runs its official per-episode gradient fine-tuning on the streams of Table 1, paired with FSSL by seed index (Table R5). FSSL keeps the highest final mean of Table 1 on all six streams.

Results. On CartPole the fine-tuned FAMLE exceeds the submitted row, 0.843 against 0.701, in each of the five seeds (p = 0.014). The remaining gap to FSSL on this stream is not significant (p = 0.186). On the other five streams fine-tuning changes FAMLE without significance (p ≥ 0.233), with FSSL higher in all five seeds (p ≤ 0.019).

Table R5. Per-step reward of the frozen final evaluation after the stream, mean ± population SD over five seeds, in the notation of Table 1. Bold marks the best mean. ▼ marks a value significantly lower than FSSL (two-sided paired t-test over seeds paired by seed index, p < 0.05). The submitted FAMLE row skipped the stream, while the fine-tuned row is the FAMLE row of the revised Table 1.

MethodCartPoleHalfCheetahAntIntersectionSlimHumanoidHopper
FSSL0.901 ± 0.0013.478 ± 0.7940.342 ± 0.0710.998 ± 0.0702.715 ± 2.0652.496 ± 0.121
FAMLE, submitted0.701 ± 0.103▼−0.009 ± 0.127▼0.044 ± 0.058▼0.418 ± 0.131▼−2.103 ± 0.595▼1.686 ± 0.125▼
FAMLE with fine-tuning0.843 ± 0.071−0.020 ± 0.112▼0.085 ± 0.027▼0.493 ± 0.236▼−1.688 ± 0.878▼1.719 ± 0.159▼

The FAMLE row of Table 1 now reports the fine-tuned runs (line 441), with the submitted means kept in Appendix B (lines 837–844).

Reply to Weakness 2: Novelty against multiple-model control and T-MCL

We thank the reviewer for this point. Our central claim does not rest on switching between fixed and adaptive models. Theorem 1 concerns one predictor that keeps learning, with no frozen copy to switch to. It bounds the planning gap of that predictor separately on revisits and at first-visit entries.

Evidence for the claim. Figure 3(a) and Section 4.2 now measure that predictor alone against its own frozen checkpoint on all six streams (lines 377–428). Under the same update a T-MCL variant shows the same two signs on HalfCheetah (Table R4). With fixed weights the drift that an update adds is zero. The asymmetry therefore belongs to learning on the stream and not to selection among fixed heads.

What prior work covers. The frozen copy and the selection rule address the first-visit side. Related Work credits both to multiple-model control and T-MCL. Related Work now also states that the stability analyses of that line cover switching among fixed and adaptive models for one time-invariant plant. Theorem 1 instead bounds each phase of a recurring stream (Section 2, page 3, lines 132–135).

Reply to Weakness 3a: Assumption of Corollary 1.1

We thank the reviewer for this point. Corollary 1.1 no longer assumes that every member selects a frozen head. Equation (9) now weights the drift term by the share ϱ\varrho of members that select learning heads, so the term vanishes at ϱ=0\varrho = 0 (page 7, lines 327–337).

The share is measured. Figure 4 now plots the heads that the planner selects during control, logged at every step. At first-visit entries members select frozen heads on 0.591 to 0.975 of the member steps (six stream means, thirty seeds), so the mean ϱ\varrho is 0.025 to 0.409 (Table R2, three of the streams in Figure 4a).

The selection rule itself. Theorem 2 in Appendix A replaces the assumption by the submitted selection rule (lines 767–819). On its selection window WW the selected head errs no more than the best frozen head, because the rule takes the lowest error over both copies. Under a Lipschitz condition its error near that window is therefore at most ∣W∣min⁡hmW(h)+Lgr\sqrt{|W| \min_h m_W(h)} + L_g r, with the minimum over frozen heads hh and no explicit drift term. Extending this bound to the planning gap needs a rollout condition that we do not measure on the streams. Proposition 2 proves a smaller planning gap with the frozen copy for stipulated scalar dynamics and heads.

Reply to Weakness 3b: Growth of the bound in Lemma 1.1

We agree with the scaling concern. The submitted bound counted one buffered pair near a point, the case νk=1\nu_k = 1, so its first term grows with the buffer at a fixed mkm_k. Lemma 1.1 now covers the points with at least νk\nu_k buffered pairs within distance rr, where more pairs nearby give a smaller bound. Under its Lipschitz condition, Equation (8) bounds the error at such a point by Nkmk/νk+Lgr\sqrt{N_k m_k/\nu_k} + L_g r. When the nearby pairs grow with the buffer, νk≥ρNk\nu_k \ge \rho N_k, the first term is at most mk/ρ\sqrt{m_k/\rho}, which does not grow with the buffer. Theorem 1 (i) uses the new form, assuming credited heads and rollouts in the covered set, with coverage and rollout conditions unmeasured (Section 3.2, page 6, lines 298–319).

Reviewer_P6HoOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes FSSL, a model-based controller whose members keep a frozen copy of their pre-trained predictions. The idea is to use pre-trained models to deal with unfamiliar environments, while continuously learning new dynamics models to handle seen environments. There are two separate situations: 1. First visit, where the current dynamics did not appear earlier, and the agent needs to adapt using pre-trained knowledge. 2. Revisit, where the dynamics appeared before, and the agent can use transitions collected during their earlier visits.

FSSL keeps two copies: the frozen copy that preserves pertinent dynamics knowledge for first visits. The learning copy updates the dynamics hypothesis using collected transitions. Then, it selects the head with the lowest prediction error on recent transitions among both copies. The MPC then plans an action sequence using the selected dynamics model, executes its first action, and replans. The learning copy is updated by MCL. The learning happens between episodes, while head selection happens during the episode.

Critical strengths
  1. The authors provide a thorough theoretical analysis of bounding the performance loss of planning with inaccurate dynamics models. They consider the planning gap, stream constraints, and the drift in unseen dynamics. By combining the previous results, the authors obtain two planning-gap bounds for revisits and first-visit entries.

  2. The authors evaluate FSSL on six environments against multiple baselines, which provides strong empirical validation. Table 1 shows that FSSL achieves the highest sample mean across all six benchmarks.

Critical weaknesses
  1. In the empirical results, although FSSL has shown better performance compared to other baselines, its improvement in the per-step reward is not that significant compared to that of the other models in most environments.

  2. In Section 3.2, the selection mechanism is defined. Each ensemble member has M frozen heads and M learning heads. At each time step, it selects the head with the lowest prediction error over the recent transitions. Part of the theoretical results largely depend on the assumption that the ensemble member selects the appropriate learning head on revisits. However, there is no actual validation of it.

Questions for authors

See weakness.

2 comments
AuthorsOpenReview ↗

Response to Reviewer P6Ho (1/2): Weakness 1

Dear Reviewer P6Ho,

Thank you for your constructive review and for acknowledging our theoretical analysis and empirical evaluation. In the revised paper, we show when the method helps over thirty seeds (W1), together with the head that each member selects during control (W2). We have addressed your concerns as follows.

Reply to Weakness 1: Size of the improvement over the baselines

We thank the reviewer for this point. Under the paired t-test, FSSL is significantly higher in 37 of the 48 comparisons of Table 1, including three of the six against T-MCL, the closest method. Beyond this final margin, our central claim is that learning on the stream can lower reward at first-visit entries while raising it on revisits, relative to the same checkpoint kept frozen (Figure 3). Our entry and revisit analysis, now extended to thirty seeds, supports this claim. Seven of the twelve entry and revisit differences are significant under the paired t-test with the sign of the claim, while nine of the twelve have that sign (Table R1). In addition, the planner selects frozen heads more often at first-visit entries than on revisits in each of thirty seeds on all six streams (Appendix Table D.7, with three streams in Figure 4). Both results support our reading of recurring dynamics, in which first visits rely on fast selection among pretrained heads and revisits benefit from slow learning. We expect this reading to help later work that faces the same trade-off between first visits and revisits. To give these two results more weight, the revised paper presents them at the start of Section 4.2 (lines 377–458), ahead of Table 1, with the abstract and the introduction revised to match.

Table R1. FSSL w/o frozen copy minus its own frozen checkpoint, per-step reward over matched episodes of seeds 0 to 29. Brackets give the 90% seed-cluster bootstrap interval. p is that of the paired t-test over seed means. Bold marks p < 0.05. W is the credit separation over seeds 0 to 29, the distance between the credit shares of two modes over the learning heads, from 0 for one split to 1 for disjoint heads. The table condenses Appendix Tables D.3 and D.4 of the revised paper.

StreamEntriespRevisitspW
CartPole−0.003 [−0.009, +0.002]0.342+0.009 [+0.007, +0.012]<0.0010.78
HalfCheetah−0.905 [−1.256, −0.550]<0.001+0.333 [+0.209, +0.455]<0.0010.73
Ant−0.089 [−0.145, −0.031]0.018−0.004 [−0.030, +0.023]0.8180.10
Intersection−0.267 [−0.366, −0.164]<0.001+0.283 [+0.232, +0.332]<0.0010.58
SlimHumanoid+0.148 [−0.729, +1.027]0.789+0.412 [−0.187, +1.010]0.2770.15
Hopper−0.078 [−0.131, −0.024]0.027−0.065 [−0.103, −0.028]0.0100.18
AuthorsOpenReview ↗

Response to Reviewer P6Ho (2/2): Weakness 2

Reply to Weakness 2: Validation of the selection mechanism

Thank you for raising this. The head that each member selects during control is now logged, in Figure 4 for three streams and Appendix Table D.7 for six (Section 4.2, lines 430–458).

Results. At first-visit entries members select frozen heads on 0.591 to 0.975 of the member steps (six stream means, thirty seeds). The range on revisits is 0.090 to 0.506. The entry share is higher in each of thirty seeds on every stream. Table R2 gives the shares by phase.

Table R2. Share of frozen heads among the heads that the planner rolls out during control. The share is averaged over steps and members within an episode, then over the episodes of a phase and over seeds 0 to 29. The last row counts the seeds with a higher share at first-visit entries than on revisits.

PhaseCartPoleHalfCheetahAntIntersectionSlimHumanoidHopper
First-visit entries0.7930.9750.7650.8350.6640.591
Rest of cycle one0.2450.3630.6740.2240.5570.436
Revisits0.2750.2670.5060.0900.3220.288
Seeds, entries > revisits30/3030/3030/3030/3030/3030/30

Consistency of the selected learning head. For FSSL with the frozen copy, we checked two things on revisits (Table R3). First, for each mode and ensemble member, the learning head selected most often stays the same across all revisit episodes for 62–87% of the mode–member pairs on CartPole, HalfCheetah, Intersection and SlimHumanoid, against 3% on Ant and 18% on Hopper. Second, the selected learning head is the one with the lowest error on the fixed set of the mode on 62% to 85% of the learning-head steps on the same four streams, against 38% on Ant and 40% on Hopper. With three learning heads the chance level of the second share is one third, which every stream exceeds in five of five seeds. In summary, on four of the six streams a member returns on revisits to one learning head, which is mostly the head of lowest error, while both patterns are weak on Ant and Hopper.

Table R3. Learning head that a member of FSSL selects on revisits, from the logged head indexes of the runs with the frozen copy, mean over seeds 0 to 4. Row 1 is the share of the learning-head steps of an episode and member on its most frequent learning head. Row 2 is the share of mode–member pairs whose most frequent learning head is the same in every revisit episode of the mode. Row 3 is the total-variation distance of learning-head use between modes. Row 4 is the share of learning-head steps on the learning head of lowest fixed-set error for the mode, with a chance level of one third. The table reproduces Appendix Table D.10 of the revised paper.

RevisitsCartPoleHalfCheetahAntIntersectionSlimHumanoidHopper
Share on the most frequent head within an episode0.9160.7620.4770.9090.7930.482
Pairs with one most frequent head over revisits0.8700.6200.0300.7470.8400.180
Distance of head use between modes0.7630.7830.0870.7100.0850.173
Steps on the head of lowest fixed-set error0.8420.6240.3770.8460.7810.398

The condition of Theorem 1 (i). Part (i) requires that slow learning credits one head per member with a mode and that the planner selects that head. On the selection side, a member returns to one learning head on four streams (Table R3). On the credit side, in FSSL w/o frozen copy the head with the most credit of a mode holds 0.70 to 0.93 of it on CartPole, HalfCheetah and Intersection over thirty seeds. It holds 0.41 and 0.45 on Ant and Hopper, where an even split gives one third. Credit is thus partly concentrated on the three streams with a revisit gain, while it is near an even split on Ant and Hopper. The joint premise on credit and selection stays unverified.

We hope that our response has addressed your concerns. As the discussion period closes soon, we will reply promptly to any further question.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to the Area Chair and All Reviewers

Dear Area Chair and Reviewers,

We thank the Area Chair and Reviewers BWmp, 8cRW and P6Ho for their careful, constructive feedback.

We appreciate the positive remarks on the problem and the approach (8cRW), on the phase analysis and the component ablations (BWmp) and on the performance against the baselines (P6Ho).

Responses to the main concerns and new experiments. Detailed replies are posted under each review, with Tables R1 to R7 numbered once across the threads.

  • Strength of the evidence (all reviewers). New experiment: seeds 5 to 29 on all six streams. Over thirty seeds the learning heads alone lower entry reward and raise revisit reward on HalfCheetah and Intersection against their frozen checkpoint, each with p < 0.001 under the paired t-test (Table R1).
  • Validation of the selection mechanism (all reviewers). New experiment: the head that each member selects during control is logged. The frozen share is higher at first-visit entries than on revisits on all six streams, in each of thirty seeds (Table R2).
  • Size of the improvement (P6Ho). FSSL is significantly higher in 37 of the 48 comparisons of Table 1, including three of the six against T-MCL.
  • Learning head on revisits (P6Ho). On four of the six streams a member returns to one learning head across revisit episodes (Table R3).
  • T-MCL without Gaussian variance (8cRW, BWmp). New experiment: this variant on all six streams, frozen and with our update. Our update raises its whole-stream reward on three streams, while we claim no margin of FSSL over the updated variant (Table R4).
  • FAMLE with adaptation (8cRW). New experiment: FAMLE with its official per-episode fine-tuning. FSSL keeps the highest final mean of Table 1 on all six streams (Table R5).
  • Lemma 1.1 and Corollary 1.1 (8cRW). Lemma 1.1 is restated, with a bound that does not grow with the buffer under a coverage condition. Corollary 1.1 no longer assumes that every member selects a frozen head.
  • Novelty (8cRW, BWmp). Multiple heads, segment credit and selection by recent error belong to T-MCL. Our claim is the asymmetry of learning on the stream between first visits and revisits, with a frozen copy that reduces the entry loss on HalfCheetah and Intersection.
  • Phase results for all methods and oracle regret (8cRW). New analysis: all eight baselines by phase. New experiment: an oracle with mode labels, whose whole-stream regret has an interval that includes zero on four of the six streams (Table R6).
  • Streams without a gain (8cRW). New measurement: on Ant and Hopper the credit of a mode is spread near evenly over the learning heads, beside the absence of a revisit gain.
  • Whole stream against the frozen checkpoint (BWmp). Over thirty seeds FSSL exceeds its frozen checkpoint over the whole stream on CartPole and Intersection, while the other four intervals include zero (Table R7).

Revisions in the paper. Blue marks the new results and the changed formal statements in the revised paper. Other rewording and removed text are not marked, so the rewritten framing is listed below.

  • Abstract and Section 1 (pages 1–2): the trade-off between first visits and revisits is stated ahead of FSSL, with three contributions.
  • Section 2 (page 3): the scope of the stability analyses of multiple-model control.
  • Section 3.2 (pages 6–7): Lemma 1.1 under coverage, Corollary 1.1 without its all-frozen assumption and the CEM optimisation term.
  • Section 4.2 (pages 7–10): the phase comparison on six streams and thirty seeds ahead of Table 1, the logged planner selections and the FAMLE row with fine-tuning.
  • Section 5 (page 10): the thirty-seed results beside the limits of the evidence.
  • Appendix A (pages 15–16): Theorem 2 and Proposition 2 on the submitted selection rule.
  • Appendix Tables D.3 and D.4 (page 20): phases and paired t-tests over thirty seeds.
  • Appendix Tables D.6 and D.7 (page 22): the T-MCL variant by phase and the planner selections.
  • Appendix Figure D.1 with Appendix Tables D.8 and D.9 (pages 23–24): all eight baselines by phase.
  • Appendix Tables D.10 and D.11 (page 25): the selected learning head on revisits and the oracle regret.
  • Figures 1 to 4 are redrawn.

The method, the title, Table 2 and every number of Table 1 outside the FAMLE row are unchanged. The submitted Table 3 is Appendix Table D.2.

We hope that our responses and revisions address the concerns raised. As the discussion period closes soon, we will reply promptly to any further question.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.