acceptodds
Under review as a conference paper at ICLR 2027

Routing-Aware Masked Decision Transformer for Task-Agnostic Continual Offline Reinforcement Learning

Abstract

Continual offline reinforcement learning (CORL) offers an efficient approach to building autonomous decision-making agents from static datasets without additional environment interactions during training. However, substantial differences in environment dynamics across tasks make it challenging to preserve previously acquired knowledge while effectively transferring it to subsequent tasks. Many existing structure-based methods manage knowledge through coarse-grained module composition, while their reliance on explicit task identities at inference limits their applicability when such information is unavailable. To address these challenges, we propose the Routing-Aware Masked Decision Transformer (RMDT), which integrates trajectory-based routing with parameter-level masking for task-agnostic CORL. Specifically, RMDT allocates subnetworks using learnable binary masks to preserve and reuse historical knowledge, while applying capacity regularization to discourage unnecessary parameter expansion. Gated low-rank expansion supplements model capacity on demand to accommodate task-specific learning requirements. To handle scenarios without explicit semantic task labels, RMDT introduces a lightweight task router during training that combines classifier predictions with Gaussian prototype matching. During inference, the router identifies the current task using multi-step probability aggregation and confidence-based locking, thereby activating the corresponding mask configuration to enable hierarchical knowledge selection. Experiments on Offline Continual World show that RMDT outperforms existing baselines, demonstrating better knowledge reuse and transfer capabilities. Code is available at here.

Primary area:
reinforcement learning
Keywords:
Decision Transformer, Continual Offline Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_WLDCOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Routing-Aware Masked Decision Transformer (RMDT) for continual offline reinforcement learning, with known task boundaries during training and no ground-truth task identifiers at task-agnostic evaluation.

RMDT combines task-specific binary masks for historical weight protection and reuse, gated low-rank expansion when remaining capacity becomes limited, and a trajectory-based router for selecting policy configurations. The router combines classifier predictions with Gaussian prototype distances, uses historical pseudo-features to mitigate forgetting, and applies temporal aggregation and episode-level task locking.

Experiments on OCW10 and OCW20 compare task-aware and task-agnostic performance, forgetting, forward transfer, and computational cost. Additional experiments examine component contributions, routing choices, and robustness to task arrival order.

Critical strengths
  1. Relevant problem and coherent design. The paper addresses task-agnostic deployment in continual offline reinforcement learning through a coherent combination of parameter management and task identification.

  2. Meaningful system-level improvements. RMDT achieves task-agnostic AP of 0.63/0.65 on OCW10/OCW20, compared with 0.52/0.53 for the strongest reported baseline.

  3. Useful comparative evidence. Baselines requiring task identifiers receive the same router, and RMDT maintains its advantage across four task orders, strengthening confidence in the reported system-level gains.

Critical weaknesses
  1. Limited methodological novelty relative to closely related work. Weight-level binary masks, selective reuse, cumulative occupancy masks, historical weight freezing, and STE-based optimization already appear in WSN (Kang et al., 2022). Moreover, the capacity regularizer in Eq. (3) has the same functional structure as HAT’s normalized penalty on newly allocated capacity (Serrà et al., 2018, Eq. (5)), although RMDT applies it to weight-level masks rather than unit-level attention. The remaining contribution is primarily the integration of these mechanisms with gated low-rank expansion and trajectory-based routing in CORL. The paper does not sufficiently establish the additional conceptual contribution of this integration.

  2. The experiments do not adequately isolate the value of the distinctive design choices. On task-agnostic OCW10, WSN with the shared router obtains 0.50 ± 0.05 AP, RMDT without low-rank expansion obtains 0.52 ± 0.07, and full RMDT obtains 0.63 ± 0.02. These results establish the benefit of the complete system but do not separate the proposed allocation rule from the effect of expansion. A WSN-style masking baseline with the same gated expansion, router, and comparable parameter budget is missing. The router also lacks an initial-state-only comparison. Finally, OCW20 repeats the same ten underlying tasks, limiting the breadth of evidence supporting the approach.

Questions for authors
  1. Relationship to prior allocation mechanisms: Please clarify the relationship between Eq. (3) and HAT’s capacity regularizer in Eq. (5). Beyond the change from unit-level attention to weight-level masks, what new allocation principle or empirical benefit does RMDT introduce?

  2. Controlled comparison with WSN: How does RMDT compare with WSN-style masks equipped with the same gated low-rank expansion and router under a comparable parameter budget? This would help determine whether the proposed allocation mechanism contributes beyond adding expansion to an existing masking approach.

  3. Contribution of trajectory information: How does an initial-state-only router compare with the full trajectory descriptor when the classifier, prototype machinery, and training budget are otherwise matched? This would clarify whether multi-step trajectory information is essential to the reported routing advantage.

3 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer WLDC [1]

Thank you for your careful review of our work. As the weaknesses you identified encompass several key points, we have broken them down and addressed each point and question individually below. Where weaknesses and questions concern the same issue, we have merged them to keep our response concise. All additional experimental results and revisions prompted by your suggestions have been incorporated into the revised manuscript.

W1: Limited methodological novelty relative to closely related work. The paper does not sufficiently establish the additional conceptual contribution of this integration.

We thank the reviewer for raising the concern about novelty, and we would like to restate what distinguishes RMDT. Our framework is essentially a dynamic hierarchical architecture for reinforcement learning across environments with different dynamics: upstream task identification drives downstream weight invocation. RMDT shares a similar mask granularity with WSN. However, WSN allocates static weights across the entire CNN for in-domain image recognition, whereas our weight allocation emphasizes rationality and scalability. Specifically, we distinguish three types of parameters under the DT architecture: frozen parameters entirely outside mask management, historically frozen parameters under mask management, and trainable parameters. While capacity regularization pushes the model to reuse historical knowledge, we further introduce newly learnable gated expansion and router-based identification, which are crucial for convergence. The contribution of this integration lies at the level of architecture and setting, rather than any single component. The main table already shows the performance gap between RMDT and WSN transferred to the DT architecture.

As for HAT, its capacity regularization acts on the task attention over hidden units, whereas ours acts directly on parameter-level binary masks. HAT therefore measures how much of the remaining unit capacity is occupied by the attention of newly added units. In contrast, RMDT measures how much of the remaining unallocated weights are occupied by newly added weights, so that the capacity constraint corresponds directly to the parameters that are actually allocated and frozen. In addition, gated low-rank expansion and trajectory-based routing are components we designed for RMDT and, to our knowledge, are the first to be effectively applied in the CORL setting. In other parts of this rebuttal (W2, Q1, and Q2), we further demonstrate through experiments the advantages of RMDT over other methods at both the architecture and component levels. We will also add an analysis and comparison with the two methods above in the main text to clarify our unique contributions.

W2: These results establish the benefit of the complete system but do not separate the proposed allocation rule from the effect of expansion. A WSN-style masking baseline with the same gated expansion, router, and comparable parameter budget is missing. The router also lacks an initial-state-only comparison. Finally, OCW20 repeats the same ten underlying tasks, limiting the breadth of evidence supporting the approach.

In the task-agnostic CW10 setting, we added a comparison using WSN-style weight allocation within a DT backbone, combined with gated expansion and the same router, to isolate the effect of the weight allocation mechanism. Please refer to Q2 for details. We also report task identification accuracy and success rate when the router uses only state information; further details are provided in Q3. CW20 primarily tests the model’s ability to reuse previously acquired knowledge. To further evaluate scalability, we selected 20 additional tasks from the original Meta-World benchmark and appended them to OCW10, constructing two extended sequences, OCW20+ and OCW30+. We also report reference upper-bound success rates obtained by independently training a full model on each task. Compared with the homogeneous image recognition tasks considered by WSN, OCW20+ and OCW30+ comprise heterogeneous reinforcement learning tasks with different dynamics and reward mechanisms. These extended sequences substantially increase the challenge relative to benchmarks commonly used in this area and provide a demanding evaluation of our method. We also include WSN-DT in these comparisons.

AuthorsOpenReview ↗

Rebuttal to Reviewer WLDC [2]

The 20 additional tasks, in training order, are listed below. They are appended after the original OCW10 sequence (tasks 1–10) as tasks 11–30: tasks 11–20 form OCW20+, and tasks 11–30 form OCW30+.

OrderTaskOrderTaskOrderTask
1hammer11reach21coffee-pull
2push-wall12drawer-open22coffee-push
3faucet-close13pick-place23dial-turn
4push-back14door-open24door-lock
5stick-pull15button-press-topdown25handle-pull
6handle-press-side16drawer-close26lever-pull
7push17peg-insert-side27plate-slide
8shelf-place18window-open28soccer
9window-close19faucet-open29stick-push
10peg-unplug-side20basketball30sweep-into

OCW20+ appends the first 10 tasks to the original CW10 sequence, while OCW30+ appends all 20. As shown in the table, RMDT maintains strong performance on both OCW20+ and OCW30+ under task-aware and task-agnostic settings, consistently outperforming WSN-DT, with less forgetting, stronger forward transfer, and smaller performance degradation as the number of tasks increases. These results support its effectiveness and scalability on long sequences of heterogeneous tasks.

SettingOCW20+ AP↑OCW20+ F↓OCW20+ FWT↑OCW30+ AP↑OCW30+ F↓OCW30+ FWT↑
Single-task DT0.82±0.020.80±0.01
WSN-DT(task-aware)0.64±0.040.00±0.010.03±0.020.56±0.050.00±0.000.02±0.02
RMDT(task-aware)0.78±0.020.00±0.000.04±0.020.74±0.030.00±0.000.05±0.02
WSN-DT(task-agnostic)0.46±0.020.14±0.040.01±0.000.38±0.030.18±0.040.00±0.01
RMDT(task-agnostic)0.62±0.030.04±0.020.04±0.020.57±0.040.06±0.020.05±0.02

Q1: Relationship to prior allocation mechanisms: Please clarify the relationship between Eq. (3) and HAT’s capacity regularizer in Eq. (5). Beyond the change from unit-level attention to weight-level masks, what new allocation principle or empirical benefit does RMDT introduce?

Compared with HAT, RMDT’s capacity regularizer serves the broader hierarchical framework by allocating capacity directly at the level of individual weight connections. Within the same hidden unit, some connections can be frozen and reused as historical knowledge, while the remaining unoccupied connections remain available for learning new tasks. The capacity cost is also computed directly from newly occupied weights. This provides finer allocation flexibility than unit-level attention, allowing shared and task-specific connections to coexist within a single unit without requiring a uniform activation decision for the entire unit.

To examine whether this flexibility yields practical benefits, we constructed an RMDT variant with neuron-level masks. Specifically, we replaced the mask scores, originally matching the shape of each weight matrix, with a score vector over the corresponding layer’s output units. Each unit’s output therefore receives a single gate, rather than allowing independent selection of its individual weight connections. Accordingly, we determined which connections to protect based on historical unit masks. We reformulated the capacity regularizer to count unoccupied units rather than unoccupied weights, aligning the penalty for newly allocated capacity with the allocation granularity. We compared average performance, forgetting, and parameter overhead while keeping the DT backbone, gated low-rank expansion mechanism, routing mechanism, and training budget unchanged. Under the same backbone and routing mechanism, with matched expansion budgets, weight-level allocation achieved higher average performance and forward transfer on both OCW10 and OCW20. These results support the practical benefits of finer allocation granularity within our framework.

Mask granularityOCW10 AP↑F↓FWT↑Params(M)OCW20 AP↑F↓FWT↑Params(M)
Unit-level0.58±0.030.04±0.020.03±0.025.90.59±0.040.04±0.020.04±0.027.0
Weight-level(RMDT)0.63±0.020.03±0.020.05±0.015.90.65±0.020.02±0.010.06±0.017.0

Q2: Controlled comparison with WSN: How does RMDT compare with WSN-style masks equipped with the same gated low-rank expansion and router under a comparable parameter budget? This would help determine whether the proposed allocation mechanism contributes beyond adding expansion to an existing masking approach.

AuthorsOpenReview ↗

Rebuttal to Reviewer WLDC [3]

In the task-agnostic OCW10 setting, we added a comparison that combines WSN-style weight allocation in a DT backbone with gated expansion and the shared router to isolate the effect of the weight-allocation mechanism. As shown in the table, RMDT still outperforms this variant. The main reason is that WSN statically selects the top-scoring weights at a fixed ratio in each layer. In contrast, RMDT selects weights through a threshold on learnable scores, so the number of weights selected in each layer can vary dynamically across tasks. RMDT also applies capacity regularization to constrain the occupation of newly added weights, encouraging knowledge reuse, and gated expansion complements this by compensating for the additional parameters some tasks require. In contrast, WSN has no capacity constraint, so it is more likely to trigger gated expansion early in the task stream, which further increases training difficulty and parameter cost.

ConfigurationOCW10 AP↑Task index at first expansionInitial parameter countFinal parameter count
WSN allocation + gated expansion + shared router0.57±0.0145.7M6.5M
RMDT allocation + gated expansion + shared router0.63±0.0295.7M5.9M

Q3: Contribution of trajectory information: How does an initial-state-only router compare with the full trajectory descriptor when the classifier, prototype machinery, and training budget are otherwise matched? This would clarify whether multi-step trajectory information is essential to the reported routing advantage.

We thank the reviewer for this question, which helps clarify the source of the routing advantage. In our response to W6 from reviewer UwMc, we showed that performance does not primarily depend on goal positions. Here, we further examine whether multi-step trajectory information itself is necessary. Since several Meta-World tasks have similar initial robot arm poses and overlapping randomized object positions, we expect task identification from the initial state alone to be challenging. As shown in the table, with the classifier, prototype mechanism, and training budget held constant, the full trajectory descriptor improves task identification accuracy from 0.61 to 0.90 and average task success rate from 0.34 to 0.63, both increases of 29 percentage points. These results indicate that the initial state alone is insufficient to match the full router’s identification accuracy. State changes and action information gathered through multi-step interaction help resolve ambiguities in the initial observation and yield practical policy performance gains. Thus, under the current evaluation setting, multi-step trajectory information is an important source of the routing advantage.

Router inputOCW10 Routing Accuracy↑OCW10 AP↑
Initial state only0.610.34±0.01
Full trajectory descriptor0.900.63±0.02

Furthermore, Figure 3 in the main text already examines how the timing of task locking affects routing accuracy and task success rate. Moderately delaying locking allows the router to use more interaction information and improve identification accuracy. However, locking too late may reduce task success due to continued policy switching. The initial-state-only comparison complements this analysis by distinguishing the contribution of multi-step trajectory information from the effect of locking timing.

Reviewer_4n5pOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper addresses task-agnostic continual offline RL, where an agent learns a sequence of tasks from fixed datasets without knowing task identity, while avoiding forgetting. The authors propose RMDT, which extends Decision Transformer with a learnable binary weight mask for parameter reuse, a gated low rank expansion for added capacity, and a trajectory based router that infers task identity from short interactions to activate the matching configuration. Evaluated on Offline Continual World against rehearsal, regularization, and structure based baselines, RMDT achieves the best average performance, lowest forgetting, and best forward transfer, with fewer parameters and FLOPs than the closest baseline, and ablations confirm each component contributes.

Critical strengths
  1. The problem setting is well chosen. Most continual offline RL work assumes task identity at test time, which is far easier than real deployment. Framing task agnostic CORL as task identification on top and parameter selection below is a clean separation of the two failure modes. This matters because the contribution is not just another architecture.

  2. The task agnostic results are consistent and efficient. RMDT reaches 0.63 and 0.65 average success on OCW10 and OCW20 versus 0.52 and 0.53 for the best baseline, with near zero forgetting and fewer parameters than CompoFormer. The router is also applied to the baselines, so the comparison is not self serving. Since task aware performance is already largely solved, this gap is the number that determines whether the direction is worth pursuing

Critical weaknesses
  1. Key comparisons are missing. Remonda et al. (2025), the work this paper cites for its task agnostic setting, is not in Table 1. The coarse grained baselines also use a different backbone, so the gain cannot be attributed to the granularity of the masks.

  2. The claim is broader than the setting. Task boundaries are assumed known and each task has its own offline dataset, so the results do not cover truly streaming settings. The forward transfer advantage is also within seed variance and does not support the stated contribution

Questions for authors

Q1. Remonda et al. (2025) is cited as closely related work on the same task-agnostic CRL setting, yet it does not appear in Table 1. Is this due to incompatibility with OCW, lack of a usable implementation, or another reason?

Q2. Given that task boundaries are assumed known, how much of the task-aware to task-agnostic performance gap in Table 1 is attributable to the router's identification error versus other factors?

Q3. Are all baselines in Table 1, including CompoFormer and VQ-CD, reimplemented on the same DT backbone as RMDT? If not, how much of the 0.75 vs 0.71 gap can be attributed to backbone differences rather than mask granularity?

Q4. On OCW20, why initialize a new mask when a task reappears instead of reusing or fine tuning the mask and routing class learned the first time?

Q5. Table 6 lists many hyperparameters. Were they tuned separately per benchmark and task order, or fixed once and reused across OCW10, OCW20, and all four task orders in Table 7?

4 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer 4n5p [1]

Thank you for your careful review of our work. As the weaknesses you identified encompass several key points, we have broken them down and addressed each point and question individually below. Where weaknesses and questions concern the same issue, we have merged them to keep our response concise. All additional experimental results and revisions prompted by your suggestions have been incorporated into the revised manuscript.

W1 (first part) & Q1: Key comparisons are missing. Remonda et al. (2025), the work this paper cites for its task agnostic setting, is not in Table 1

UBER is a closely related work that we overlooked in our literature review. We fully recognize its pioneering contribution to the task-agnostic setting. As the authors' code was not publicly available, we made our best effort to reproduce UBER, tune its hyperparameters, and adapt it to our offline evaluation setting. However, the adapted implementation performed poorly. We believe this is due to several differences between the two settings. First, UBER relies on online interaction: when the model's predictions are unreliable, it replans and collects new experience, thereby progressively improving its dynamics model. In our offline setting, the training data is fixed, and the model cannot actively acquire new experience to compensate for prediction errors, which is particularly problematic because OOD is severe. Second, UBER's experience selection mechanism is tightly coupled with online trajectory execution, relying on the discrepancy between the originally predicted future reward distribution and the distribution re-predicted from the actual new state. When transferred to fixed offline trajectories, the actions in the data do not necessarily match those chosen by the current planner, so the feedback loop among planning, interaction, and experience selection in the original method cannot be fully preserved. Finally, UBER performs multi-step planning with a learned dynamics model; with insufficient coverage in offline data, candidate action sequences may drift away from the training distribution, causing prediction errors to accumulate over multi-step rollouts and degrading action selection. Because of this mismatch between the two settings, we believe including UBER in the main table would not constitute a fair comparison with the original method. We therefore present the results below and will add the comparison with UBER, along with a discussion of the results, to the appendix.

SettingMethodOCW10 AP↑F↓FWT↑OCW20 AP↑F↓FWT↑
Task-awareUBER0.390.160.080.410.180.04
Task-awareRMDT0.750.000.050.760.000.06
Task-agnosticUBER0.390.160.080.410.180.04
Task-agnosticRMDT0.630.030.050.650.020.06

W1 (second part) & Q3: The coarse grained baselines also use a different backbone, so the gain cannot be attributed to the granularity of the masks. Are all baselines in Table 1 reimplemented on the same DT backbone as RMDT? If not, how much of the 0.75 vs 0.71 gap can be attributed to backbone differences rather than mask granularity?

The baselines in Table 1 that use the DT architecture include WSN, PM, A-GEM, EWC, MAS, CompoFormer, LoRA, and PackNet, ensuring a fair comparison. For VQ-CD, the strongest competing baseline, we retain its original U-Net diffusion backbone and weight selection mechanism, while configuring its model capacity and training budget to match those of RMDT. Specifically, we start with a hidden dimension of 256 and adjust the network width according to the actual parameter count, conducting multiple rounds of tuning to obtain the best performance for VQ-CD. We also implemented a DT-based variant of VQ-CD, but it performed worse than the original version; we report its results below. One possible explanation is that VQ-CD's weight selection and assembly mechanism was designed around the one-dimensional convolutional structure of U-Net, whose parameter grouping and feature processing differ from those of DT's attention and feed-forward networks. Moreover, replacing the diffusion backbone with a standard DT changes both the training objective and the inference procedure, which may also have contributed to the poor performance.

MethodOCW10 AP↑ (task-aware)OCW20 AP↑ (task-aware)
VQ-CD0.71±0.030.74±0.04
VQ-CD (DT adaptation)0.59±0.020.63±0.04
RMDT0.75±0.010.76±0.01
AuthorsOpenReview ↗

Rebuttal to Reviewer 4n5p [2]

We acknowledge that the remaining baselines (OER, VCL, and UCL) were adapted from their original implementations and tuned, and we report the best results within the configurations explored. We did not convert these methods to a common DT backbone because they rely heavily on their architectural designs, which are not readily compatible with DT’s autoregressive formulation. We also believe that weight-level masking is well suited to Transformer architectures: it selectively preserves and reuses historical weights within attention projection matrices and feed-forward layers, without requiring allocation at the level of entire attention heads, neurons, or modules. This enables more flexible combinations of shared and task-specific knowledge. To directly examine the effect of allocation granularity, we added a comparison between neuron-level and weight-level masks using the same DT backbone, keeping the routing mechanism, training budget, and expansion budget identical. Under the task-agnostic setting, weight-level masking achieves AP scores of 0.63 and 0.65 on OCW10 and OCW20, respectively, compared with 0.58 and 0.59 for neuron-level masking. These results support the practical benefit of finer allocation granularity within our framework.

Mask granularityOCW10 AP↑OCW20 AP↑
Neuron-level0.58±0.030.59±0.04
Weight-level (RMDT)0.63±0.020.65±0.02

W2: The claim is broader than the setting. Task boundaries are assumed known and each task has its own offline dataset, so the results do not cover truly streaming settings. The forward transfer advantage is also within seed variance and does not support the stated contribution

We thank the reviewer for the thoughtful comment. Since RMDT neither retains past real data nor trains an additional dynamics model, we do relax the requirement on task boundaries. We consider this setting reasonably acceptable in real-world scenarios, since humans can distinguish different tasks through perceived differences in dynamics. We will revise the relevant statements in the paper to clarify the scope of applicability of our method. Regarding the limitation of using only three random seeds in the original experiments, which may allow chance factors to obscure the advantage, we extended the comparison between RMDT and the strongest baseline, VQ-CD, to eight random seeds and recomputed the forward transfer metric. As shown in the table below, RMDT still achieves a higher average FWT in the extended experiments, providing further empirical support for its forward transfer advantage.

SettingMetricOCW10OCW20
Task-awareVQ-CD FWT↑0.03±0.010.05±0.01
Task-awareRMDT FWT↑0.05±0.010.06±0.01
Task-agnosticVQ-CD FWT↑0.02±0.010.00±0.01
Task-agnosticRMDT FWT↑0.05±0.010.06±0.01

Q2: Given that task boundaries are assumed known, how much of the task-aware to task-agnostic performance gap in Table 1 is attributable to the router's identification error versus other factors?

We thank the reviewer for this suggestion. The performance gap does arise from routing, but it cannot be attributed solely to incorrect final task identification. During evaluation, before the task identity is determined and locked, the model must infer it from gradually accumulated interaction information. During this period, it may temporarily select an inappropriate policy configuration and execute actions that deviate from the current task’s goal. These early actions alter subsequent states and trajectories. Even if the model eventually identifies the task correctly, earlier trajectory deviations or an insufficient remaining interaction budget may prevent successful completion. Thus, the gap reflects both losses from locking onto an incorrect task identity and the accumulated effects of temporary misidentification before locking.

To isolate the loss attributable to incorrect task identification at locking, we added a comparison in which, when the router locks its prediction, the model replaces the predicted identity with the oracle task identity. As shown in the table, providing the correct identity at locking improves performance, but does not match the performance achieved when the task identity is available from the first step. This indicates that the gap stems not only from incorrect identities after locking, but also from policy choices before locking and their lasting effects on subsequent trajectories.

SettingOCW10 AP↑OCW20 AP↑
Original routing0.63±0.020.65±0.02
Oracle ID at locking0.69±0.010.71±0.01
Oracle ID from start0.75±0.010.76±0.01
AuthorsOpenReview ↗

Rebuttal to Reviewer 4n5p [3]

Q4: On OCW20, why initialize a new mask when a task reappears instead of reusing or fine tuning the mask and routing class learned the first time?

This is an issue we carefully considered when designing our method. In practical scenarios, particularly without explicit task labels, we generally cannot directly determine whether an incoming task matches a previously learned one. Reusing or fine-tuning a previous task’s configuration would therefore require an additional task identification mechanism during training. Misidentification at this stage can have more lasting consequences than at evaluation, where it affects only the current episode:

  1. If a new task is incorrectly identified as a previous task and its existing parameters are reused without further training, the model forgoes learning from the current data stream. Under the continual learning setting, these missed data may no longer be accessible.
  2. Fine-tuning a previous task’s configuration requires unfreezing its parameters. An incorrect identification could compromise learning of the new task, degrade performance on the previous task, and potentially affect other tasks that share those parameters.
  3. If fine-tuning instead allocates additional capacity while preserving historical parameters, this becomes conceptually similar to RMDT. It would also require handling mask reassignment, potentially increasing computational cost and reducing the method’s simplicity.

Our design prioritizes the continual arrival of novel tasks, while CW20 provides an additional test of the model’s ability to reuse previously acquired knowledge. Even when a new configuration is initialized for a recurring task, adaptive masking and capacity allocation allow the model to reuse historical weights while allocating relatively little additional capacity. This can enable convergence in fewer training steps to performance comparable to, or even exceeding, that achieved when the task was first encountered. This ability to reuse prior knowledge without explicitly identifying recurring tasks during training is a central motivation behind our design.

Q5: Were hyperparameters tuned separately or fixed once and reused across OCW10, OCW20, and all four task orders in Table 7?

Yes. To simplify practical deployment, we use the same hyperparameters across all tasks and settings.

Dear Reviewer 4n5p,

We sincerely thank you again for your valuable feedback, which has helped us improve the quality of our work. As the discussion deadline is approaching, we kindly request you to review our rebuttal. We have also uploaded a revised PDF of the paper, addressing your concerns to the best of our ability. We are happy to address any remaining concerns. Thank you again for your time and effort!

Best regards,
Authors

Reviewer_UwMcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies continual offline reinforcement learning: an agent learns a sequence of tasks, each from a static dataset, without environment interaction and without access to past-task data, and must act without a task identity at test time. The proposed method combines three components on top of a decision transformer backbone. First, parameter-level binary masks: each task learns a binary mask over backbone weights via a straight-through estimator; weights occupied by earlier tasks are gradient-blocked and their values restored after every optimizer step, while new tasks may reuse old weights in their forward pass; a capacity regularizer limits the fraction of remaining capacity allocated per task. Second, gated low-rank expansion: when remaining capacity drops below twenty percent, low-rank modules are added and composed through trainable scalar gates. Third, trajectory-based routing: short interaction windows are summarized by a descriptor (initial state, displacement, mean state, mean action); per-task Gaussian prototypes and a classifier trained with Gaussian pseudo-replay jointly infer the task, which is locked once confidence exceeds a threshold.

Critical strengths

Removing the task identity at test time is the practically important setting for continual reinforcement learning deployment, and here the method clearly outperforms all twelve baselines (0.63 versus 0.52 for the best baseline on the ten-task benchmark; 0.65 versus 0.53 on the twenty-task benchmark). The routing analysis is unusually thorough: prototype-count sweeps, a confusion matrix, an ablation showing that Gaussian generative pseudo-replay is essential to the classifier (average performance collapses from 0.63 to 0.37 without it), and the mildly counterintuitive locking result.

Critical weaknesses
  1. Near-zero forgetting is guaranteed by construction: occupied weights have their gradients masked and their values restored after every optimizer step, so reporting forgetting of 0.00 as evidence of "effective knowledge retention" is circular — the interesting question is whether the shrinking free capacity retains plasticity, which is never directly probed (no per-task upper bound is provided). Relatedly, the capacity regularizer as written is a single global fraction in the unit interval divided by millions of remaining parameters; with the stated coefficient of 1e-4 its per-parameter gradient is on the order of 1e-10, effectively zero, yet the ablation attributes a forgetting change from 0.03 to 0.10 to it. Either the equation, the coefficient, or the ablation is misreported, and this needs clarification.
  2. There is no naive fine-tuning lower bound and no single-task or joint multitask upper bound, so an average performance of 0.75 has no interpretable reference; two recent, directly relevant methods are discussed in the text but absent from the tables; and all identity-requiring baselines borrow the authors' own router in the task-agnostic setting, with no alternative router or oracle-identity gap analysis. Moreover, the benchmark's observations include goal positions, and the routing descriptor contains the initial and mean state, so tasks are plausibly separable from the first frame; the reported ninety percent routing accuracy and the "task-agnostic" framing should be read with this leakage in mind, and no goal-masked control is provided. Finally, with only three seeds, several task-aware margins (0.75 versus 0.71; 0.76 versus 0.74) are within or near noise.
Questions for authors
  1. Please clarify the capacity regularizer: its exact implementation, the effective gradient magnitude per parameter, and whether the stated coefficient of 1e-4 is the one used to produce the ablation.
5 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer UwMc [1]

Thank you for your careful review of our work. As the weaknesses you identified encompass several key points, we have broken them down and addressed each point and question individually below. Where weaknesses and questions concern the same issue, we have merged them to keep our response concise. All additional experimental results and revisions prompted by your suggestions have been incorporated into the revised manuscript.

W1: Reporting forgetting of 0.00 as evidence of "effective knowledge retention" is circular — the interesting question is whether the shrinking free capacity retains plasticity, which is never directly probed (no per-task upper bound is provided).

We thank the reviewer for pointing out this flaw in our argument. Our architecture indeed guarantees zero forgetting by design, but this alone does not logically imply effective knowledge retention. We will revise the corresponding statements in the main text and provide empirical evidence to support this claim. Under the task-aware setting, we independently trained a DT with the same model size as in the main experiments on each task, updating all parameters, as an upper-bound reference for success rate. We report RMDT’s performance on each task after completing the CW10 sequence. We also independently trained a smaller DT on each task, updating all parameters and matching its parameter count to the capacity used by RMDT for that task, including the added LoRA parameters, as a capacity-matched baseline. As shown in the table, RMDT closely approaches the full-parameter fine-tuning reference across the ten tasks despite using less capacity per task, and even outperforms it on some tasks. These results provide further evidence that RMDT preserves plasticity through knowledge retention and reuse, offsetting the adverse effects of having fewer trainable parameters and enabling better decisions on the current task.

Settinghammerpush-wallfaucet-closepush-backstick-pullhandle-press-sidepushshelf-placewindow-closepeg-unplug-sideAvg.
Full-size single-task DT (upper bound)0.970.591.000.670.631.000.501.001.000.770.81
RMDT1.000.501.000.500.800.900.200.801.000.800.75
Capacity-matched single-task DT (lower bound)0.790.210.920.300.350.860.170.600.920.440.56

To further examine the effect of capacity regularization on plasticity, we varied the capacity regularization coefficient across 1e-1, 1e-2, 1e-3, and 1e-4. Larger coefficients allocate fewer parameters. For most of the evaluated values, performance remains relatively stable, supporting the model's ability to maintain plasticity within this range. When the coefficient is set to 1e-1, the average task success rate drops modestly, suggesting that knowledge reused from previous tasks cannot fully compensate for the limited trainable capacity available to accommodate task-specific differences. Given that nearly half of the model parameters remain unused, we consider this drop a reasonable trade-off.

Capacity regularization coefficientOCW10 AP↑ (task-aware)Final remaining base capacity (%)
1e-10.6549.8
1e-20.7233.8
1e-30.7419.0
1e-40.7517.1

Since the tasks in OCW20 overlap with those in OCW10, we selected ten new tasks from Meta-World. We appended them to the original OCW10 sequence as tasks 11–20, forming OCW20+, to better examine RMDT's plasticity and scalability over longer task sequences. This 20-task stream, with diverse dynamics and reward structures, is more challenging than the benchmarks used by the baseline methods. We report RMDT's success rates on these additional tasks during continual learning, alongside the single-task upper bound. As shown in the table, RMDT remains competitive on these previously unseen tasks, demonstrating strong plasticity even in this more demanding setting.

Settingreachdrawer-openpick-placedoor-openbutton-press-topdowndrawer-closepeg-insert-sidewindow-openfaucet-openbasketballAvg.
Single-task DT0.970.890.530.860.940.980.610.930.880.670.83
RMDT(task-aware)0.960.870.570.830.920.970.580.940.860.620.81
AuthorsOpenReview ↗

Rebuttal to Reviewer UwMc [2]

W2 & Q1: Either the equation, the coefficient, or the ablation is misreported, and this needs clarification. Please clarify the capacity regularizer: its exact implementation, the effective gradient magnitude per parameter, and whether the stated coefficient of 1e-4 is the one used to produce the ablation.

We sincerely thank the reviewer for carefully reading our work. The capacity regularizer is implemented exactly as described in the main text, and the coefficient of 1e-4 was indeed used in the ablation experiments. We apologize for the data-recording error in Appendix Table 10. After rerunning the ablation with three different random seeds and checking the historical experiment logs, we found that the forgetting score and its standard deviation were recorded incorrectly. The correct result is 0.04±0.01. We will correct the table and revise the accompanying analysis. Since the architecture protects historical policy parameters, the residual performance degradation in the task-agnostic setting is primarily attributable to potential routing errors. The corrected table is provided below.

VariantAP↑F↓FWT↑
w/o Weight Reuse0.58±0.060.08±0.040.02±0.02
w/o Capacity Regularization0.60±0.050.04±0.010.02±0.02
Full RMDT0.63±0.020.03±0.020.05±0.01

W3: There is no naive fine-tuning lower bound and no single-task or joint multitask upper bound.

In our response to W1, we reported an upper bound obtained by training a full-size model independently on each task. We further provide a naive sequential fine-tuning lower bound on CW10, which is shared across the task-aware and task-agnostic settings. Following the training setup of HarmoDT [1], we also include joint multi-task training on the same ten tasks as CW10 as an additional reference. All results are averaged over three random seeds. As shown in the table, RMDT substantially outperforms naive sequential fine-tuning and approaches the performance of the full-size single-task DT, providing empirical support for its ability to learn new tasks and maintain plasticity throughout continual learning.

SettingBenchmarkAP↑F↓
Naive sequential fine-tuningCW100.280.46
Single-task DTCW100.81-
Joint multi-task DTMT100.84-

[1] Harmodt: Harmony multi-task decision trans former for offline reinforcement learning

W4: Two recent, directly relevant methods are discussed in the text but absent from the tables.

During the early stages of our experiments, we carefully selected baseline methods. We sought methods with publicly available code and fully reproducible results to ensure fair comparisons, as we needed to adapt them to the task-agnostic continual learning setting, which their original evaluation setups generally did not support. We are not entirely sure which two papers the reviewer is referring to, but speculate that they may be two of CoD [2], HTAC [3], and UBER [4]. Since VQ-CD and CoD are works by the same authors, and the VQ-CD paper reports that CoD performs worse than VQ-CD, we initially considered VQ-CD sufficient to represent the best performance of this line of work and therefore did not include CoD as a baseline. HTAC and UBER had not released their code. We did make careful attempts to reproduce their reported performance during the early stages of our experiments, but some discrepancies remained. To avoid misunderstandings, we did not include them as baselines. In the table below, we report the results of adapting all three methods, against which our method still retains a performance advantage. We will include these additional baselines in the appendix and explain the reproduction outcomes out of respect for the original authors’ work.

SettingMethodOCW10 AP↑F↓FWT↑OCW20 AP↑F↓FWT↑
Task-awareCoD0.680.040.020.680.050.03
Task-awareHTAC0.700.030.040.710.040.05
Task-awareUBER0.390.160.080.410.180.04
Task-awareRMDT0.750.000.050.760.000.06
Task-agnosticCoD0.470.130.010.500.100.02
Task-agnosticHTAC0.520.090.020.550.080.03
Task-agnosticUBER0.390.160.080.410.180.04
Task-agnosticRMDT0.630.030.050.650.020.06

[2] Continual Diffuser (CoD): Mastering Continual Offline Reinforcement Learning with Experience Rehearsal
[3] HTAC: Hierarchical Task-Aware Composition for Continual Offline Reinforcement Learning
[4] Uncertainty-Based Experience Replay for Task-Agnostic Continual Reinforcement Learning

AuthorsOpenReview ↗

Rebuttal to Reviewer UwMc [3]

W5: All identity-requiring baselines borrow the authors' own router in the task-agnostic setting, with no alternative router or oracle-identity gap analysis.

We thank the reviewer for the suggestion. In the main experiments, we used the same routing mechanism across methods to control variables as much as possible and adapted baselines that originally required explicit task identifiers to the task-agnostic setting, allowing their continual learning capabilities to be compared under consistent conditions. Since our router is plug-and-play, adaptation issues generally do not cause additional performance degradation.  

To investigate the advantages of our proposed routing mechanism, we additionally compared it with the reconstruction-based routing in Expert Gate (CVPR 2017) [5] and the VAE-based distribution-matching routing in CoPE (ICML 2026) [6], both of which are highly flexible and among the best-performing routing mechanisms to date. We evaluated task identification accuracy on PackNet, VQ-CD, and RMDT, where RA denotes task identification accuracy, and AP denotes average task success rate. As shown in the table, the RMDT router achieves higher identification accuracy and task success rates across all three policy methods. It narrows the performance gap to Oracle ID, supporting its effectiveness in task-agnostic settings and its applicability across methods.

MethodExpert Gate(RA/AP)CoPE(RA/AP)RMDT(RA/AP)Oracle ID(RA/AP)
PackNet79%/0.4280%/0.4688%/0.48100%/0.64
VQ-CD76%/0.3882%/0.4287%/0.44100%/0.71
RMDT(Ours)83%/0.5285%/0.5790%/0.63100%/0.75

Meanwhile, the Oracle-Identity setting corresponds to model performance when task identities are known, and thus serves as a 100% upper bound for routing accuracy. Without access to explicit task identities, our router captures informative trajectory features to approach this upper bound, demonstrating the effectiveness of the routing mechanism. We further address the reviewer's concerns about potential information leakage in our response to W6.

[5] Expert Gate: Lifelong Learning with a Network of Experts [6] CoPE: Continual Probe-guided Expansion for Large Vision-Language Models

W6: The benchmark's observations include goal positions, and the routing descriptor contains the initial and mean state, so tasks are plausibly separable from the first frame; the reported ninety percent routing accuracy and the "task-agnostic" framing should be read with this leakage in mind, and no goal-masked control is provided.

We thank the reviewer for carefully reading our work. This question is particularly important for readers to correctly understand our evaluation benchmark. In our main experiments, the goal position is indeed included in the last three dimensions of the observation (state). However, the goal position distributions for the same task often differ substantially between training and evaluation. In contrast, those of different tasks in the training set overlap considerably. Consequently, a router that identifies tasks solely based on goal positions observed during training may fail to generalize to evaluation, suggesting that the potential impact of information leakage through goal positions is limited. To further illustrate this, we report statistics of goal positions in the CW10 training and evaluation sets, covering 2,000 offline training trajectories and 10 evaluation resets per task.

ComparisonMinimum Center DistanceMedian Center DistanceMaximum Center Distance
Between different tasks in the training set (45 pairs)23.713251.128669.564
Between training and evaluation for the same task (10 tasks)69.470311.147638.249

To directly examine whether goal information makes task identification easier, we conducted additional experiments in which the last three dimensions of the observation were either (1) set to zero or (2) removed entirely during both training and evaluation. We also included an initial-state-only baseline while keeping the classifier, prototype mechanism, and training budget unchanged. As shown in the table, masking goal positions does not noticeably decrease either routing accuracy or task success rate, whereas using only the initial state substantially reduces both metrics. These results indicate that the router does not primarily rely on explicit goal position information in the current setting and further support the practical contribution of multi-step interaction information to both task identification and policy performance.

Routing VariantRouting Accuracy↑Average Success Rate↑
(1) Zeroed Goal0.890.62±0.02
(2) Removed Goal0.900.63±0.01
Initial State Only0.610.34±0.01
Full Trajectory Descriptor (RMDT)0.900.63±0.02

Comments on the paper

2 comments

We sincerely thank the reviewer for kindly raising the score. We have incorporated all these results into the revised manuscript as suggested.

AuthorsOpenReview ↗

Summary of Manuscript Revisions

Dear reviewers,

We sincerely thank you for your constructive feedback and helpful suggestions. In response, we have revised the manuscript and highlighted the key additions in blue for ease of review. Below is a summary of the main updates.

Additional Experiments and Analyses

Plasticity and reference baselines
[Appendix E.7, Tables 14–15] We added full-size and capacity-matched single-task references, naive sequential fine-tuning, and joint multi-task training to evaluate plasticity and contextualize performance. [Reviewer UwMc, W1 & W3]

Longer sequences of distinct tasks
[Appendix E.8, Tables 16–18] We added OCW20+ and OCW30+ evaluations to examine scalability and plasticity as new tasks continue to arrive. [Reviewer UwMc, W1; Reviewer WLDC, W2]

Controlled allocation and mask-granularity comparisons
[Section 5.2, Table 5; Appendix E.9, Tables 19–20] We added controlled comparisons of allocation mechanisms and mask granularities, together with an analysis of expansion behavior. [Reviewer 4n5p, W1 & Q3; Reviewer WLDC, W1, W2, Q1 & Q2]

Additional baselines and backbone adaptation
[Appendix E.10, Tables 21–22] We added implementations or adaptations of CoD, HTAC, and UBER, together with a DT-based adaptation of VQ-CD and a discussion of adaptation limitations. [Reviewer UwMc, W4; Reviewer 4n5p, W1, Q1 & Q3]

Alternative routers and oracle task identities
[Appendix E.11, Tables 23–24] We added comparisons with alternative routing mechanisms and oracle task identities, and analyzed the effects of identification errors and policy execution before task locking. [Reviewer UwMc, W5; Reviewer 4n5p, Q2]

Goal-information and trajectory-input controls
[Section 5.2, Table 3; Appendix E.11, Table 25] We added controls with goal coordinates zeroed or removed, plus an initial-state-only comparison, to examine potential information leakage and the contribution of trajectory information. [Reviewer UwMc, W6; Reviewer WLDC, W2 & Q3]

Evaluation across eight random seeds
[Appendix E.12, Tables 26–27] We expanded the RMDT–VQ-CD comparison to eight seeds and reported updated performance and forward-transfer statistics. [Reviewer UwMc, W7; Reviewer 4n5p, W2]

Corrections and Clarifications

Capacity regularization and corrected ablation results
[Section 4.1; Appendix E.3, Tables 11–12] We clarified the capacity regularizer, corrected the ablation results, and added a coefficient sweep to examine its effect on performance and remaining capacity. [Reviewer UwMc, W1, W2 & Q1; Reviewer WLDC, Q1]

Task-boundary assumptions and recurring tasks
[Sections 3 and 5.1] We clarified the assumption of known task boundaries during offline training and explained how we allocate new masks when tasks recur. [Reviewer 4n5p, W2 & Q4]

We provide further details in our point-by-point responses and thank the reviewers again for helping us improve the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.