- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Routing-Aware Masked Decision Transformer (RMDT) for continual offline reinforcement learning, with known task boundaries during training and no ground-truth task identifiers at task-agnostic evaluation.
RMDT combines task-specific binary masks for historical weight protection and reuse, gated low-rank expansion when remaining capacity becomes limited, and a trajectory-based router for selecting policy configurations. The router combines classifier predictions with Gaussian prototype distances, uses historical pseudo-features to mitigate forgetting, and applies temporal aggregation and episode-level task locking.
Experiments on OCW10 and OCW20 compare task-aware and task-agnostic performance, forgetting, forward transfer, and computational cost. Additional experiments examine component contributions, routing choices, and robustness to task arrival order.
Critical strengths
-
Relevant problem and coherent design. The paper addresses task-agnostic deployment in continual offline reinforcement learning through a coherent combination of parameter management and task identification.
-
Meaningful system-level improvements. RMDT achieves task-agnostic AP of 0.63/0.65 on OCW10/OCW20, compared with 0.52/0.53 for the strongest reported baseline.
-
Useful comparative evidence. Baselines requiring task identifiers receive the same router, and RMDT maintains its advantage across four task orders, strengthening confidence in the reported system-level gains.
Critical weaknesses
-
Limited methodological novelty relative to closely related work. Weight-level binary masks, selective reuse, cumulative occupancy masks, historical weight freezing, and STE-based optimization already appear in WSN (Kang et al., 2022). Moreover, the capacity regularizer in Eq. (3) has the same functional structure as HAT’s normalized penalty on newly allocated capacity (Serrà et al., 2018, Eq. (5)), although RMDT applies it to weight-level masks rather than unit-level attention. The remaining contribution is primarily the integration of these mechanisms with gated low-rank expansion and trajectory-based routing in CORL. The paper does not sufficiently establish the additional conceptual contribution of this integration.
-
The experiments do not adequately isolate the value of the distinctive design choices. On task-agnostic OCW10, WSN with the shared router obtains 0.50 ± 0.05 AP, RMDT without low-rank expansion obtains 0.52 ± 0.07, and full RMDT obtains 0.63 ± 0.02. These results establish the benefit of the complete system but do not separate the proposed allocation rule from the effect of expansion. A WSN-style masking baseline with the same gated expansion, router, and comparable parameter budget is missing. The router also lacks an initial-state-only comparison. Finally, OCW20 repeats the same ten underlying tasks, limiting the breadth of evidence supporting the approach.
Questions for authors
-
Relationship to prior allocation mechanisms: Please clarify the relationship between Eq. (3) and HAT’s capacity regularizer in Eq. (5). Beyond the change from unit-level attention to weight-level masks, what new allocation principle or empirical benefit does RMDT introduce?
-
Controlled comparison with WSN: How does RMDT compare with WSN-style masks equipped with the same gated low-rank expansion and router under a comparable parameter budget? This would help determine whether the proposed allocation mechanism contributes beyond adding expansion to an existing masking approach.
-
Contribution of trajectory information: How does an initial-state-only router compare with the full trajectory descriptor when the classifier, prototype machinery, and training budget are otherwise matched? This would clarify whether multi-step trajectory information is essential to the reported routing advantage.
3 comments
Rebuttal to Reviewer WLDC [2]
The 20 additional tasks, in training order, are listed below. They are appended after the original OCW10 sequence (tasks 1–10) as tasks 11–30: tasks 11–20 form OCW20+, and tasks 11–30 form OCW30+.
| Order | Task | Order | Task | Order | Task |
|---|---|---|---|---|---|
| 1 | hammer | 11 | reach | 21 | coffee-pull |
| 2 | push-wall | 12 | drawer-open | 22 | coffee-push |
| 3 | faucet-close | 13 | pick-place | 23 | dial-turn |
| 4 | push-back | 14 | door-open | 24 | door-lock |
| 5 | stick-pull | 15 | button-press-topdown | 25 | handle-pull |
| 6 | handle-press-side | 16 | drawer-close | 26 | lever-pull |
| 7 | push | 17 | peg-insert-side | 27 | plate-slide |
| 8 | shelf-place | 18 | window-open | 28 | soccer |
| 9 | window-close | 19 | faucet-open | 29 | stick-push |
| 10 | peg-unplug-side | 20 | basketball | 30 | sweep-into |
OCW20+ appends the first 10 tasks to the original CW10 sequence, while OCW30+ appends all 20. As shown in the table, RMDT maintains strong performance on both OCW20+ and OCW30+ under task-aware and task-agnostic settings, consistently outperforming WSN-DT, with less forgetting, stronger forward transfer, and smaller performance degradation as the number of tasks increases. These results support its effectiveness and scalability on long sequences of heterogeneous tasks.
| Setting | OCW20+ AP↑ | OCW20+ F↓ | OCW20+ FWT↑ | OCW30+ AP↑ | OCW30+ F↓ | OCW30+ FWT↑ |
|---|---|---|---|---|---|---|
| Single-task DT | 0.82±0.02 | 0.80±0.01 | ||||
| WSN-DT(task-aware) | 0.64±0.04 | 0.00±0.01 | 0.03±0.02 | 0.56±0.05 | 0.00±0.00 | 0.02±0.02 |
| RMDT(task-aware) | 0.78±0.02 | 0.00±0.00 | 0.04±0.02 | 0.74±0.03 | 0.00±0.00 | 0.05±0.02 |
| WSN-DT(task-agnostic) | 0.46±0.02 | 0.14±0.04 | 0.01±0.00 | 0.38±0.03 | 0.18±0.04 | 0.00±0.01 |
| RMDT(task-agnostic) | 0.62±0.03 | 0.04±0.02 | 0.04±0.02 | 0.57±0.04 | 0.06±0.02 | 0.05±0.02 |
Q1: Relationship to prior allocation mechanisms: Please clarify the relationship between Eq. (3) and HAT’s capacity regularizer in Eq. (5). Beyond the change from unit-level attention to weight-level masks, what new allocation principle or empirical benefit does RMDT introduce?
Compared with HAT, RMDT’s capacity regularizer serves the broader hierarchical framework by allocating capacity directly at the level of individual weight connections. Within the same hidden unit, some connections can be frozen and reused as historical knowledge, while the remaining unoccupied connections remain available for learning new tasks. The capacity cost is also computed directly from newly occupied weights. This provides finer allocation flexibility than unit-level attention, allowing shared and task-specific connections to coexist within a single unit without requiring a uniform activation decision for the entire unit.
To examine whether this flexibility yields practical benefits, we constructed an RMDT variant with neuron-level masks. Specifically, we replaced the mask scores, originally matching the shape of each weight matrix, with a score vector over the corresponding layer’s output units. Each unit’s output therefore receives a single gate, rather than allowing independent selection of its individual weight connections. Accordingly, we determined which connections to protect based on historical unit masks. We reformulated the capacity regularizer to count unoccupied units rather than unoccupied weights, aligning the penalty for newly allocated capacity with the allocation granularity. We compared average performance, forgetting, and parameter overhead while keeping the DT backbone, gated low-rank expansion mechanism, routing mechanism, and training budget unchanged. Under the same backbone and routing mechanism, with matched expansion budgets, weight-level allocation achieved higher average performance and forward transfer on both OCW10 and OCW20. These results support the practical benefits of finer allocation granularity within our framework.
| Mask granularity | OCW10 AP↑ | F↓ | FWT↑ | Params(M) | OCW20 AP↑ | F↓ | FWT↑ | Params(M) |
|---|---|---|---|---|---|---|---|---|
| Unit-level | 0.58±0.03 | 0.04±0.02 | 0.03±0.02 | 5.9 | 0.59±0.04 | 0.04±0.02 | 0.04±0.02 | 7.0 |
| Weight-level(RMDT) | 0.63±0.02 | 0.03±0.02 | 0.05±0.01 | 5.9 | 0.65±0.02 | 0.02±0.01 | 0.06±0.01 | 7.0 |
Q2: Controlled comparison with WSN: How does RMDT compare with WSN-style masks equipped with the same gated low-rank expansion and router under a comparable parameter budget? This would help determine whether the proposed allocation mechanism contributes beyond adding expansion to an existing masking approach.
Rebuttal to Reviewer WLDC [3]
In the task-agnostic OCW10 setting, we added a comparison that combines WSN-style weight allocation in a DT backbone with gated expansion and the shared router to isolate the effect of the weight-allocation mechanism. As shown in the table, RMDT still outperforms this variant. The main reason is that WSN statically selects the top-scoring weights at a fixed ratio in each layer. In contrast, RMDT selects weights through a threshold on learnable scores, so the number of weights selected in each layer can vary dynamically across tasks. RMDT also applies capacity regularization to constrain the occupation of newly added weights, encouraging knowledge reuse, and gated expansion complements this by compensating for the additional parameters some tasks require. In contrast, WSN has no capacity constraint, so it is more likely to trigger gated expansion early in the task stream, which further increases training difficulty and parameter cost.
| Configuration | OCW10 AP↑ | Task index at first expansion | Initial parameter count | Final parameter count |
|---|---|---|---|---|
| WSN allocation + gated expansion + shared router | 0.57±0.01 | 4 | 5.7M | 6.5M |
| RMDT allocation + gated expansion + shared router | 0.63±0.02 | 9 | 5.7M | 5.9M |
Q3: Contribution of trajectory information: How does an initial-state-only router compare with the full trajectory descriptor when the classifier, prototype machinery, and training budget are otherwise matched? This would clarify whether multi-step trajectory information is essential to the reported routing advantage.
We thank the reviewer for this question, which helps clarify the source of the routing advantage. In our response to W6 from reviewer UwMc, we showed that performance does not primarily depend on goal positions. Here, we further examine whether multi-step trajectory information itself is necessary. Since several Meta-World tasks have similar initial robot arm poses and overlapping randomized object positions, we expect task identification from the initial state alone to be challenging. As shown in the table, with the classifier, prototype mechanism, and training budget held constant, the full trajectory descriptor improves task identification accuracy from 0.61 to 0.90 and average task success rate from 0.34 to 0.63, both increases of 29 percentage points. These results indicate that the initial state alone is insufficient to match the full router’s identification accuracy. State changes and action information gathered through multi-step interaction help resolve ambiguities in the initial observation and yield practical policy performance gains. Thus, under the current evaluation setting, multi-step trajectory information is an important source of the routing advantage.
| Router input | OCW10 Routing Accuracy↑ | OCW10 AP↑ |
|---|---|---|
| Initial state only | 0.61 | 0.34±0.01 |
| Full trajectory descriptor | 0.90 | 0.63±0.02 |
Furthermore, Figure 3 in the main text already examines how the timing of task locking affects routing accuracy and task success rate. Moderately delaying locking allows the router to use more interaction information and improve identification accuracy. However, locking too late may reduce task success due to continued policy switching. The initial-state-only comparison complements this analysis by distinguishing the contribution of multi-step trajectory information from the effect of locking timing.
Rebuttal to Reviewer WLDC [1]
Thank you for your careful review of our work. As the weaknesses you identified encompass several key points, we have broken them down and addressed each point and question individually below. Where weaknesses and questions concern the same issue, we have merged them to keep our response concise. All additional experimental results and revisions prompted by your suggestions have been incorporated into the revised manuscript.
W1: Limited methodological novelty relative to closely related work. The paper does not sufficiently establish the additional conceptual contribution of this integration.
We thank the reviewer for raising the concern about novelty, and we would like to restate what distinguishes RMDT. Our framework is essentially a dynamic hierarchical architecture for reinforcement learning across environments with different dynamics: upstream task identification drives downstream weight invocation. RMDT shares a similar mask granularity with WSN. However, WSN allocates static weights across the entire CNN for in-domain image recognition, whereas our weight allocation emphasizes rationality and scalability. Specifically, we distinguish three types of parameters under the DT architecture: frozen parameters entirely outside mask management, historically frozen parameters under mask management, and trainable parameters. While capacity regularization pushes the model to reuse historical knowledge, we further introduce newly learnable gated expansion and router-based identification, which are crucial for convergence. The contribution of this integration lies at the level of architecture and setting, rather than any single component. The main table already shows the performance gap between RMDT and WSN transferred to the DT architecture.
As for HAT, its capacity regularization acts on the task attention over hidden units, whereas ours acts directly on parameter-level binary masks. HAT therefore measures how much of the remaining unit capacity is occupied by the attention of newly added units. In contrast, RMDT measures how much of the remaining unallocated weights are occupied by newly added weights, so that the capacity constraint corresponds directly to the parameters that are actually allocated and frozen. In addition, gated low-rank expansion and trajectory-based routing are components we designed for RMDT and, to our knowledge, are the first to be effectively applied in the CORL setting. In other parts of this rebuttal (W2, Q1, and Q2), we further demonstrate through experiments the advantages of RMDT over other methods at both the architecture and component levels. We will also add an analysis and comparison with the two methods above in the main text to clarify our unique contributions.
W2: These results establish the benefit of the complete system but do not separate the proposed allocation rule from the effect of expansion. A WSN-style masking baseline with the same gated expansion, router, and comparable parameter budget is missing. The router also lacks an initial-state-only comparison. Finally, OCW20 repeats the same ten underlying tasks, limiting the breadth of evidence supporting the approach.
In the task-agnostic CW10 setting, we added a comparison using WSN-style weight allocation within a DT backbone, combined with gated expansion and the same router, to isolate the effect of the weight allocation mechanism. Please refer to Q2 for details. We also report task identification accuracy and success rate when the router uses only state information; further details are provided in Q3. CW20 primarily tests the model’s ability to reuse previously acquired knowledge. To further evaluate scalability, we selected 20 additional tasks from the original Meta-World benchmark and appended them to OCW10, constructing two extended sequences, OCW20+ and OCW30+. We also report reference upper-bound success rates obtained by independently training a full model on each task. Compared with the homogeneous image recognition tasks considered by WSN, OCW20+ and OCW30+ comprise heterogeneous reinforcement learning tasks with different dynamics and reward mechanisms. These extended sequences substantially increase the challenge relative to benchmarks commonly used in this area and provide a demanding evaluation of our method. We also include WSN-DT in these comparisons.