- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes WAM-MTRL-VE, a framework for purely online multi-task reinforcement learning that integrates world-model-based policy optimization with collection-matched task updates. The method uses a shared task-conditioned representation for the actor and latent world model, enabling differentiable multi-step value expansion to guide policy learning. It further introduces collection-matched stochastic optimization to align per-task update frequencies with data collection rates, complemented by intrinsic exploration rewards and success replay. Experiments on Meta-World MT10 and MT50 achieve 96.2% success with 1M interactions and 82.1% success with 12M interactions, respectively. Ablation studies investigate the effects of update scheduling, value expansion, and success replay, demonstrating the effectiveness of the overall framework.
Critical strengths
-
Well-motivated problem formulation and systematic evaluation. The paper identifies a meaningful mismatch between task-wise data collection and optimization frequencies in online MTRL, supported by a clear UTD analysis. Comparisons across multiple update schedules, including controlled experiments with and without intrinsic rewards, provide useful evidence for the proposed optimization strategy.
-
Strong empirical performance with limited online interactions. The method achieves competitive results on Meta-World MT10 (96.2% success with 1M interactions) and MT50 (82.1% with 12M interactions), without offline pretraining or demonstrations. These results demonstrate the practical potential of the framework, although the MT50 evaluation is based on a single training run.
Critical weaknesses
-
Limited novelty and insufficient isolation of the core contributions. Most components build on established techniques, while Collection-Matched Optimization primarily introduces a relatively straightforward task-sampling strategy. Its improvement over the compute-matched Uniform Random-Single baseline is modest (97.3% vs. 93.3% on MT10), and the absence of a model-free Collection-Matched baseline makes it difficult to isolate the contribution of World–Action Coupling.
-
Insufficient experimental reliability and comparability. All MT50 results are based on a single training run, with reported standard deviations reflecting evaluation variability rather than training stability. Furthermore, baseline comparisons involve different observation modalities, episode horizons, and interaction budgets, limiting the strength of the claimed performance and sample-efficiency advantages.
Questions for authors
please see weakness.
10 comments
Response to Weakness 2 — MT50 reliability and baseline comparability
Thank you for raising the limited number of MT50 training seeds and the effect of selecting the best checkpoint. We extended the original budget of 12M environment interactions to 15M and evaluated five independent runs at both checkpoints. Our primary metric is the unsmoothed evaluation at exactly 15M steps. We evaluate the policy directly in the simulator, counting an episode as successful if the simulator signals success at any step. Figure 8 of the revision shows the learning curves for all 50 tasks.
| Training seed | Success at 15M (%) | Best observed success through 15M (%) |
|---|---|---|
| 42 | 89.6 | 91.0 |
| 66 | 86.6 | 88.6 |
| 100 | 84.4 | 85.8 |
| 585 | 87.2 | 88.4 |
| 2026 | 85.2 | 85.4 |
| Mean ± sample SD | 86.6 ± 2.0 | 87.8 ± 2.3 |
Across these five runs, mean success was 76.7% ± 3.7% at the original 12M budget and 86.6% ± 2.0% at the fixed 15M endpoint. Taking the highest observed score within each run's 15M budget gave 87.8% ± 2.3%, raising the mean by 1.24 percentage points.
If both evaluations count success at any step, a success in the first 150 steps still counts when the same trajectory is evaluated over a longer window. With the policy, initial state, goal and environment held fixed, extra steps can give an episode that has yet to succeed another opportunity. Our 150-step cap therefore confers no advantage from extra time within an episode.
The published MT50 baselines vary in their observations, episode horizons and interaction budgets. The values below are those reported in Table 1 of the revision. Our row uses the fixed 15M endpoint.
| MT50 method | Observations | Episode horizon (steps) | Environment interactions | Success (%) |
|---|---|---|---|---|
| MTSAC | State | 150 | 20M | 49.3 ± 1.5 |
| CARE | State | 150 | 20M | 50.8 ± 1.0 |
| PaCo | State | 150 | 20M | 57.3 ± 1.3 |
| MOORE | State | 150 | 100M | 72.9 ± 3.3 |
| MoW | Image | 500 | 15M | 74.5 ± 1.1 |
| MOORE + HyMTRL | State | 200 | 75M | 86.2 ± 3.4 |
| WAM-MTRL-VE | State | 150 | 15M | 86.6 ± 2.0 |
MoW matches our interaction budget but uses image observations and a longer horizon. The four baselines with our observation modality and horizon use larger budgets. These comparisons place our MT50 result in context. Establishing a sample-efficiency ranking would require evaluating the methods under the same protocol.
Response to Questions
Thank you for these questions. We checked the manuscript and appendix. Our paper introduces no theorem on covering numbers or imitation-learning error propagation, and its analysis does not use the PAC assumptions described in the questions. Section 3.1 describes two aspects of coupling. The actor and world model share a task-conditioned context, and predicted rewards guide policy optimization through differentiable multi-step rollouts. Model-side parameters remain fixed along this actor-gradient path. The three questions therefore seem to concern a different theoretical and experimental setting. We would be happy to clarify any aspect of our method or experiments that you had in mind.
Response to Weakness 1 — What collection matching and actor-side value expansion each contribute
Thank you for suggesting a model-free baseline with the same collection-matched schedule. Our method addresses two problems in online multi-task reinforcement learning. First, tasks acquire data at different rates. We allocate updates in proportion to collection so that each task's update count tracks its contribution of new transitions. Second, we update the actor parameters by backpropagating gradients from predicted rewards through short, multi-step world-model rollouts. The model-side parameters stay fixed along this gradient path. We added a five-seed model-free SAC control and compared it with the existing no-VE ablation on the same seeds to distinguish these contributions.
All four configurations below use collection matching, update one task at a time, and share the MT10 task set, the 1M-step interaction budget and the evaluation protocol. Model-free SAC retains the four-expert task representation and success replay. Its critic loss trains the representation. It uses no dynamics model, reward predictor, imagined rollouts or intrinsic rewards. For each seed (42, 66, 100, 585 and 2026), we use the logged mean success across tasks at exactly 1M steps. The overall column gives the mean and sample standard deviation across seeds. Each task column gives that task's five-seed mean success.
Value of the full framework. With the schedule held fixed, full WAM outperformed model-free SAC on all five seeds. Its paired advantage in final success was 44.2 ± 6.3 percentage points. The gap was concentrated on difficult tasks. SAC recorded zero successes on Pick-Place and Peg-Insert-Side at the endpoint on every seed. Full WAM reached 82% and 96% on those tasks. Together with Push, they accounted for 26.0 percentage points of the overall gap, or 58.8%. Even with the same collection-matched schedule, the model-free policy struggled to learn these tasks within the interaction budget. These results support the value of the full framework beyond scheduling alone.
Performance with environment rewards alone. We also compared SAC with WAM retaining its world model and multi-step VE, with intrinsic rewards disabled for both agents. At 1M steps, WAM reached 90.4% ± 3.1% mean success and SAC reached 52.0% ± 5.4%. WAM led on every seed, with a paired advantage of 38.4 ± 6.6 percentage points. On Pick-Place and Peg-Insert-Side, WAM reached 72% and 92% respectively, while SAC recorded zero successes on both tasks on all five seeds. WAM therefore retains a substantial advantage on these difficult tasks when both agents learn from environment rewards alone.
Incremental contribution of VE. The no-VE ablation retains world-model training, intrinsic rewards, success replay and collection matching. It removes only the VE term from the actor objective. Restoring VE improved final success by 4, 7, 3, 2 and 1 percentage points across the five seeds, giving a paired gain of 3.4 ± 2.3 percentage points. Without VE, WAM already reached 82% on Pick-Place and 94% on Peg-Insert-Side. Restoring VE left Pick-Place at 82% and raised Peg-Insert-Side by 2 points. The 44.2-point gain measures the advantage of the full framework over model-free SAC. The 3.4-point gain measures what VE adds to final multi-task success with the other components held fixed.
Contribution of the sampling rule. The original uniform single-task result of 93.3% ± 4.9% summarized three runs. We could not recover all individual records, so we reran this baseline under the original protocol on the five seeds above. Both schedules use the full agent and process one task minibatch every two environment steps. The last column gives the range across the ten tasks after averaging each task's measured updates per new transition over seeds, from buffer readiness through 1M steps.
With the same total number of updates, collection matching kept each task close to the reference of 0.5 updates per new transition derived in Section 3.4. It improved final success on every seed, with a paired gain of 5.6 ± 2.7 percentage points.