acceptodds
Under review as a conference paper at ICLR 2027

World–Action Coupling and Collection-Matched Optimization for Online Multi-Task Reinforcement Learning

Abstract

Purely online multi-task reinforcement learning (MTRL) couples a global optimization clock to task-local data streams. With all-task updates at a fixed global frequency, task-wise update-to-data ratios can increase with the number of tasks. We introduce **WAM-MTRL-VE**, which couples latent future prediction to policy learning while matching task optimization to online data acquisition. The actor and an ensemble world model share a task-conditioned representation, and short imagined rollouts provide multi-step reward gradients to the actor through a frozen-model computation path. At each optimization tick, collection-matched stochastic optimization samples one replay-ready task according to the collection weights. Once all replay buffers are ready, matching update and collection probabilities recovers the nominal single-task update-to-data ratio in expectation while interleaving tasks. The entire agent is trained from scratch using online experience, without demonstrations or offline pretraining. On Meta-World, WAM-MTRL-VE reaches success on MT10 after 1M total interactions per run, averaged over five training seeds. On MT50, a checkpoint selected from one 12M-interaction training run achieves success across five independently seeded evaluation suites.

Primary area:
reinforcement learning
Keywords:
Multi-Task Reinforcement Learning, World Action Models, Differentiable Value Expansion, Collection-Matched Optimization, Intrinsic Motivation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_NosHOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes WAM-MTRL-VE, a framework for purely online multi-task reinforcement learning that integrates world-model-based policy optimization with collection-matched task updates. The method uses a shared task-conditioned representation for the actor and latent world model, enabling differentiable multi-step value expansion to guide policy learning. It further introduces collection-matched stochastic optimization to align per-task update frequencies with data collection rates, complemented by intrinsic exploration rewards and success replay. Experiments on Meta-World MT10 and MT50 achieve 96.2% success with 1M interactions and 82.1% success with 12M interactions, respectively. Ablation studies investigate the effects of update scheduling, value expansion, and success replay, demonstrating the effectiveness of the overall framework.

Critical strengths
  1. Well-motivated problem formulation and systematic evaluation. The paper identifies a meaningful mismatch between task-wise data collection and optimization frequencies in online MTRL, supported by a clear UTD analysis. Comparisons across multiple update schedules, including controlled experiments with and without intrinsic rewards, provide useful evidence for the proposed optimization strategy.

  2. Strong empirical performance with limited online interactions. The method achieves competitive results on Meta-World MT10 (96.2% success with 1M interactions) and MT50 (82.1% with 12M interactions), without offline pretraining or demonstrations. These results demonstrate the practical potential of the framework, although the MT50 evaluation is based on a single training run.

Critical weaknesses
  1. Limited novelty and insufficient isolation of the core contributions. Most components build on established techniques, while Collection-Matched Optimization primarily introduces a relatively straightforward task-sampling strategy. Its improvement over the compute-matched Uniform Random-Single baseline is modest (97.3% vs. 93.3% on MT10), and the absence of a model-free Collection-Matched baseline makes it difficult to isolate the contribution of World–Action Coupling.

  2. Insufficient experimental reliability and comparability. All MT50 results are based on a single training run, with reported standard deviations reflecting evaluation variability rather than training stability. Furthermore, baseline comparisons involve different observation modalities, episode horizons, and interaction budgets, limiting the strength of the claimed performance and sample-efficiency advantages.

Questions for authors

please see weakness.

10 comments
AuthorsOpenReview ↗

Response to Weakness 1 — What collection matching and actor-side value expansion each contribute

Thank you for suggesting a model-free baseline with the same collection-matched schedule. Our method addresses two problems in online multi-task reinforcement learning. First, tasks acquire data at different rates. We allocate updates in proportion to collection so that each task's update count tracks its contribution of new transitions. Second, we update the actor parameters by backpropagating gradients from predicted rewards through short, multi-step world-model rollouts. The model-side parameters stay fixed along this gradient path. We added a five-seed model-free SAC control and compared it with the existing no-VE ablation on the same seeds to distinguish these contributions.

All four configurations below use collection matching, update one task at a time, and share the MT10 task set, the 1M-step interaction budget and the evaluation protocol. Model-free SAC retains the four-expert task representation and success replay. Its critic loss trains the representation. It uses no dynamics model, reward predictor, imagined rollouts or intrinsic rewards. For each seed (42, 66, 100, 585 and 2026), we use the logged mean success across tasks at exactly 1M steps. The overall column gives the mean and sample standard deviation across seeds. Each task column gives that task's five-seed mean success.

Agent under collection matchingOverall MT10 success at 1MPushPick-PlacePeg-Insert-Side
Model-free SAC52.0% ± 5.4%10%0%0%
WAM-MTRL-VE, intrinsic rewards off, VE enabled90.4% ± 3.1%82%72%92%
WAM-MTRL-VE, VE disabled92.8% ± 3.3%88%82%94%
Full WAM-MTRL-VE96.2% ± 1.6%92%82%96%

Value of the full framework. With the schedule held fixed, full WAM outperformed model-free SAC on all five seeds. Its paired advantage in final success was 44.2 ± 6.3 percentage points. The gap was concentrated on difficult tasks. SAC recorded zero successes on Pick-Place and Peg-Insert-Side at the endpoint on every seed. Full WAM reached 82% and 96% on those tasks. Together with Push, they accounted for 26.0 percentage points of the overall gap, or 58.8%. Even with the same collection-matched schedule, the model-free policy struggled to learn these tasks within the interaction budget. These results support the value of the full framework beyond scheduling alone.

Performance with environment rewards alone. We also compared SAC with WAM retaining its world model and multi-step VE, with intrinsic rewards disabled for both agents. At 1M steps, WAM reached 90.4% ± 3.1% mean success and SAC reached 52.0% ± 5.4%. WAM led on every seed, with a paired advantage of 38.4 ± 6.6 percentage points. On Pick-Place and Peg-Insert-Side, WAM reached 72% and 92% respectively, while SAC recorded zero successes on both tasks on all five seeds. WAM therefore retains a substantial advantage on these difficult tasks when both agents learn from environment rewards alone.

Incremental contribution of VE. The no-VE ablation retains world-model training, intrinsic rewards, success replay and collection matching. It removes only the VE term from the actor objective. Restoring VE improved final success by 4, 7, 3, 2 and 1 percentage points across the five seeds, giving a paired gain of 3.4 ± 2.3 percentage points. Without VE, WAM already reached 82% on Pick-Place and 94% on Peg-Insert-Side. Restoring VE left Pick-Place at 82% and raised Peg-Insert-Side by 2 points. The 44.2-point gain measures the advantage of the full framework over model-free SAC. The 3.4-point gain measures what VE adds to final multi-task success with the other components held fixed.

Contribution of the sampling rule. The original uniform single-task result of 93.3% ± 4.9% summarized three runs. We could not recover all individual records, so we reran this baseline under the original protocol on the five seeds above. Both schedules use the full agent and process one task minibatch every two environment steps. The last column gives the range across the ten tasks after averaging each task's measured updates per new transition over seeds, from buffer readiness through 1M steps.

Full agent, MT10 update scheduleSuccess at 1M (%)Normalized AUC (%)Range of task-wise mean UTD
Uniform single-task90.6 ± 3.450.25 ± 3.050.249–1.409
Collection-matched96.2 ± 1.652.76 ± 1.750.489–0.515

With the same total number of updates, collection matching kept each task close to the reference of 0.5 updates per new transition derived in Section 3.4. It improved final success on every seed, with a paired gain of 5.6 ± 2.7 percentage points.

AuthorsOpenReview ↗

Response to Weakness 2 — MT50 reliability and baseline comparability

Thank you for raising the limited number of MT50 training seeds and the effect of selecting the best checkpoint. We extended the original budget of 12M environment interactions to 15M and evaluated five independent runs at both checkpoints. Our primary metric is the unsmoothed evaluation at exactly 15M steps. We evaluate the policy directly in the simulator, counting an episode as successful if the simulator signals success at any step. Figure 8 of the revision shows the learning curves for all 50 tasks.

Training seedSuccess at 15M (%)Best observed success through 15M (%)
4289.691.0
6686.688.6
10084.485.8
58587.288.4
202685.285.4
Mean ± sample SD86.6 ± 2.087.8 ± 2.3

Across these five runs, mean success was 76.7% ± 3.7% at the original 12M budget and 86.6% ± 2.0% at the fixed 15M endpoint. Taking the highest observed score within each run's 15M budget gave 87.8% ± 2.3%, raising the mean by 1.24 percentage points.

If both evaluations count success at any step, a success in the first 150 steps still counts when the same trajectory is evaluated over a longer window. With the policy, initial state, goal and environment held fixed, extra steps can give an episode that has yet to succeed another opportunity. Our 150-step cap therefore confers no advantage from extra time within an episode.

The published MT50 baselines vary in their observations, episode horizons and interaction budgets. The values below are those reported in Table 1 of the revision. Our row uses the fixed 15M endpoint.

MT50 methodObservationsEpisode horizon (steps)Environment interactionsSuccess (%)
MTSACState15020M49.3 ± 1.5
CAREState15020M50.8 ± 1.0
PaCoState15020M57.3 ± 1.3
MOOREState150100M72.9 ± 3.3
MoWImage50015M74.5 ± 1.1
MOORE + HyMTRLState20075M86.2 ± 3.4
WAM-MTRL-VEState15015M86.6 ± 2.0

MoW matches our interaction budget but uses image observations and a longer horizon. The four baselines with our observation modality and horizon use larger budgets. These comparisons place our MT50 result in context. Establishing a sample-efficiency ranking would require evaluating the methods under the same protocol.

AuthorsOpenReview ↗

Response to Questions

Thank you for these questions. We checked the manuscript and appendix. Our paper introduces no theorem on covering numbers or imitation-learning error propagation, and its analysis does not use the PAC assumptions described in the questions. Section 3.1 describes two aspects of coupling. The actor and world model share a task-conditioned context, and predicted rewards guide policy optimization through differentiable multi-step rollouts. Model-side parameters remain fixed along this actor-gradient path. The three questions therefore seem to concern a different theoretical and experimental setting. We would be happy to clarify any aspect of our method or experiments that you had in mind.

Reviewer_GQsBOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the mismatch between data collection and optimization in purely online MTRL. The authors observe that conventional all-task updates can increase the per-task UTD ratio as the number of tasks grows. They propose collection-matched optimization, where each update selects one task according to the task collection distribution. The method also uses a task-conditioned latent world model to generate short imagined rollouts for actor learning, without updating the world model through this learning path. They further incorporate intrinsic exploration and success replay. The experiments cover Meta-World MT10 and MT50, with ablations examining update scheduling, value expansion, and success replay.

Critical strengths
  1. The paper identifies a concrete optimization issue in purely online MTRL. With conventional all-task updates, the per-task UTD ratio can increase with the number of tasks. The proposed collection-matched update rule is straightforward, and its motivation is clear. The update probabilities are matched to the task collection probabilities. I think this is one of the main strengths of this paper.
  2. The MT10 experiments directly target the paper's scheduling claim. They compare collection-matched, uniform random-single, freshness-gated, and naive joint updates under the same environment-interaction budget, and also compare collection-matched and naive joint schedules with intrinsic rewards disabled. These comparisons help isolate the effect of update scheduling, rather than just showing that the full method outperforms the baselines.
Critical weaknesses
  1. The MT50 result is based on only one independent training run, which makes it difficult to assess its robustness. The reported 82.1%±0.5%82.1\% \pm 0.5\% comes from evaluating a selected checkpoint from that run on five independently seeded evaluation suites, so the reported variation does not reflect training-seed variance. Since MT50 is the main evidence for performance with a larger number of tasks, I would like to see results from multiple independent training runs.
  2. The ablations are useful, but some results leave the added benefit of specific design choices less clear. For scheduling, collection-matched updates achieve 97.3±0.697.3 \pm 0.6, while the simpler uniform random-single schedule already achieves 93.3±4.993.3 \pm 4.9; the much larger difference is relative to naive joint updates. Thus, the experiments provide stronger evidence for the disadvantage of all-task updates than for the specific benefit of matching qτq_\tau to pτp_\tau. The value-expansion results are also positive, but it is less clear how consistent this benefit is. On MT10, the full variant achieves 97.397.3%, compared with 94.794.7% without VE. However, the AUC estimates show considerable variability across the three training seeds, making the consistency of the improvement less clear. On MT50, the VE ablations are based on single training runs. Additional results would help clarify how consistently these design choices improve performance.
Questions for authors
  1. Could the authors report MT50 results from more independent training runs, preferably including performance at the fixed 12M-interaction endpoint rather than only at the selected best checkpoint? The reported 82.1%±0.5%82.1\% \pm 0.5\% reflects variation across evaluation suites from a single run, not across training seeds. More runs would help assess the robustness of the MT50 result.
  2. For the comparison between collection-matched and uniform random-single updates, could the authors provide per-seed results or evaluate both methods with additional training seeds? Since both methods perform one task update per optimization tick, this would help determine whether matching qτq_\tau to pτp_\tau provides a consistent benefit over uniform task sampling. It would also be useful to know how non-uniform pτp_\tau is during training and the actual per-task UTD values.
6 comments
AuthorsOpenReview ↗

Response to Weakness 1 — Variation across MT50 training runs

Thank you for pointing out the limitation of a single MT50 training run. The original 82.1% ± 0.5% measured variation across five sets of evaluation goals for the same selected checkpoint. We had completed only one training run by the submission deadline. We now report the unsmoothed success rate at two fixed budgets for each of five independent training seeds.

MT50 training seedSuccess at 12M (%)Success at 15M (%)
4273.289.6
6676.486.6
10073.084.4
58580.687.2
202680.285.2
Mean ± SD across training seeds76.7 ± 3.786.6 ± 2.0

All five policies improved between 12M and 15M steps. At 15M, final success ranged from 84.4% to 89.6%. The 2.0-point standard deviation measures variation across independently trained policies at that fixed endpoint.

AuthorsOpenReview ↗

Response to Weakness 2 — Contributions of task sampling and value expansion

Task sampling. Uniform single-task sampling provides a direct control for the allocation rule. Both schedules use the full agent, the same collection rule and the same budget of one task minibatch every two environment steps. We reran the uniform baseline on the same five seeds as collection matching: 42, 66, 100, 585 and 2026. All gains below are computed within seeds before averaging. Each ± denotes the sample standard deviation across training seeds or their paired differences.

MT10 metricCollection-matchedUniform single-taskPaired gain (percentage points)
Success at 1M steps96.2% ± 1.6%90.6% ± 3.4%+5.6 ± 2.7
Normalized AUC over 0–1M steps52.76% ± 1.75%50.25% ± 3.05%+2.51 ± 1.47

Collection matching improved final success by 5, 10, 3, 4 and 6 percentage points in seed order. This comparison isolates the benefit of choosing the update task according to its collection weight. The comparison with naive joint updates also changes the number of task minibatches processed per tick, so that comparison alone cannot isolate the sampling effect.

Actor-side value expansion (VE). We added seeds 66 and 585 to the no-VE ablation. Both variants retain collection matching, world-model training, intrinsic rewards and success replay. The only change is whether predicted multi-step rewards contribute to the actor objective.

MT10 metricMulti-step VENo VEPaired gain (percentage points)
Success at 1M steps96.2% ± 1.6%92.8% ± 3.3%+3.4 ± 2.3
Normalized AUC over 0–1M steps52.76% ± 1.75%51.91% ± 3.39%+0.85 ± 5.05

Restoring VE improved final success on all five MT10 seeds by 4, 7, 3, 2 and 1 percentage points in the same seed order. Its effect on full-run AUC was less consistent, with a positive gain on two seeds. The five-seed evidence therefore supports a consistent endpoint benefit on MT10, while the AUC gain varies across runs.

Controlled MT50 ablations. We have now extended both comparisons to MT50 on three shared seeds: 42, 100 and 2026. All configurations use six experts, the same collection rule and evaluation protocol, a fixed budget of 15M interactions, and one task minibatch every two environment steps. Uniform updating changes only the distribution used to select an update task. The no-VE control retains collection matching, world-model training, intrinsic rewards and success replay, and removes the predicted multi-step reward term from the actor objective.

MT50 configurationSuccess at 15M (%)Normalized AUC over 0–15M (%)Paired success gain of full WAM (percentage points)
Full WAM-MTRL-VE86.4 ± 2.855.76 ± 1.44—
Uniform single-task updates67.1 ± 4.955.75 ± 2.49+19.3 ± 6.3
No VE68.1 ± 2.855.37 ± 3.41+18.3 ± 4.0

All three rows use the same three seeds, including the corresponding subset of the full-agent runs. The five-seed benchmark result of 86.6% ± 2.0% is reported separately. Relative to uniform updating, the paired endpoint gains were 24.8, 20.6 and 12.4 percentage points in seed order. Relative to no-VE, they were 22.0, 18.8 and 14.0 points. Thus, both collection matching and actor-side VE improve the endpoint on every paired MT50 seed.

The AUCs are similar across configurations. At 5M steps, the full agent averaged 49.7% success, compared with 56.5% for uniform updating and 57.7% for no-VE. By 10M steps, the corresponding means were 75.3%, 65.6% and 65.3%. These results show that the endpoint gains arise from stronger progress later in training. They support the contributions of collection matching and VE to final performance without implying uniformly faster learning.

The task-level results also show that the gains extend beyond a few tasks. Across the same three seeds, the full agent improved mean endpoint success on 38 of 50 tasks relative to uniform updating and on 43 relative to no-VE. For example, Disassemble reached 90.0% under the full agent, 0.0% under uniform updating and 16.7% without VE. The revision reports all task-level endpoints and the paired learning curves in the MT50 appendix.

AuthorsOpenReview ↗

Response to Question 1 — MT50 performance at fixed checkpoints

We evaluated each of five independently trained policies at the original budget of 12M interactions and the extended budget of 15M. The table also gives the highest evaluation score observed within each run's 15M budget. We evaluate the policy directly in the simulator. An episode counts as successful if the simulator signals success at any step.

Training seedSuccess at 12M (%)Success at 15M (%)Best observed success through 15M (%)
4273.289.691.0
6676.486.688.6
10073.084.485.8
58580.687.288.4
202680.285.285.4
Mean ± sample SD76.7 ± 3.786.6 ± 2.087.8 ± 2.3

Mean success was 76.7% ± 3.7% at 12M and 86.6% ± 2.0% at 15M. Within these same five new runs, selecting the highest observed score separately for each run gave 87.8% ± 2.3%, raising the mean by 1.24 percentage points relative to the fixed 15M endpoint. We use the fixed 15M endpoint as the primary result in the revised paper. Figure 8 of the revision shows the learning curves for all 50 tasks.

Reviewer_23pSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates two interrelated problems in purely online multi-task reinforcement learning: how to use future state predictions to guide policy optimization, and how to align the task optimization frequency with the online data collection frequency. The authors propose WAM-MTRL-VE, which builds a shared task-conditional representation, latent world model, and policy network based on SAC. The method comprises two core mechanisms: First, it employs an orthogonal expert representation similar to MOORE to construct latent task-conditional representations, enabling the actor, critic, and ensemble world model to share task context; it also introduces short-term imagined rollouts via actor-side value expansion, allowing predicted rewards to be backpropagated to the policy through a differentiable dynamical model while freezing the world model parameters. Second, the authors propose collection-matched stochastic optimization. Traditional all-task joint updates at a fixed global frequency may cause task-level UTD to increase as the number of tasks grows. The proposed method maintains single-task UTD in the expected sense when the buffer is ready and the collection and update probabilities are consistent. The framework also combines ensemble disagreement with RND to generate intrinsic exploration rewards and uses capacity-gated replays of successful trajectories to reinforce successful behaviors. Overall, the paper aims to demonstrate that online multi-task reinforcement learning requires not only the sharing of prediction and control representations but also the coordination of optimization frequency and experience acquisition rates at the task level to improve learning performance under a limited interaction budget.

Critical strengths
  1. The authors have clearly identified the mismatch between task-level data collection and optimization frequencies in online MTRL and provided targeted experimental evidence. In Section 3.4, Equation (6), the authors establish the relationship between the task data collection probability, the task update probability, and UTD, and propose a collection-matched scheduling method. Table 2 further provides a comparison of scheduling methods under the same training seed. The significance of these results lies in the fact that the paper not only proposes a new scheduling rule but also preliminarily demonstrates that this optimization of frequency control may be a critical design factor in online multi-task learning, rather than merely an additional implementation technique.

  2. Combines predictions from a differentiable world model with a shared multi-task policy and provides component-level verification. Sections 3.1–3.2 enable short-term predicted rewards to directly influence policy gradients through a shared task-conditional representation and actor-side value expansion, while freezing the world model parameters. In Table 3, the full multi-step VE configuration achieves 97.3% ± 0.6% on MT10, higher than the 94.7% ± 2.1% achieved without VE. These results provide directional support for utilizing world models from online learning to offer additional signals for strategy optimization.

Critical weaknesses
  1. The main experiments on MT50 are not yet sufficient to fully support the method’s robust advantages on large-scale task sets. Table 1 reports that WAM-MTRL-VE achieved 82.1% ± 0.5% on MT50, but this result comes from the best checkpoint in a single 12M-interaction training run, which was subsequently retested on five random evaluation sets. Therefore, the reported standard deviation reflects variations in the evaluation environment rather than variations between independent training runs (Section 4.1, Table 1, Appendix G). This is particularly important because world models, intrinsic rewards, and multi-task policy optimization may all be sensitive to training randomness. At this stage, the paper cannot sufficiently determine whether the results from a single training run are representative of the method’s general performance. Additionally, some baselines in Table 1 employ different interaction budgets, episode horizons, and observation modalities; for example, MOORE uses 100M steps, MoW uses image observations with H=500, and MOORE+HyMTRL uses H=200. Consequently, while Table 1 is suitable for illustrating the relative standing of various prior results, it is insufficient to support a rigorous performance ranking under the same protocol and budget. MT50 serves as crucial evidence that the paper’s method can be scaled to larger task sets. If this result proves unstable across different training seeds, or if the advantage significantly diminishes under the same protocol, it would undermine the assessment of the method’s overall effectiveness and significance.

  2. The paper provides experimental evidence that the scheduling mechanism affects the final performance, but has not yet demonstrated that this effect is primarily achieved through the attenuation of the exploration signal. As shown in Table 2, the “collection-matched” and “naive joint” approaches not only have different task-wise UTDs but also exhibit significant differences in the computational load per update. The paper explicitly states that the “naive joint” approach processes approximately N times as many task minibatches at each optimization step. Furthermore, although the “naive joint” method has a lower success rate at the 1M endpoint, it achieves a higher AUC when intrinsic reward is enabled (56.2% ± 2.3% vs. 53.8% ± 1.5%), indicating that different scheduling methods involve different trade-offs between early learning speed and final performance. The authors also provide a uniform random-single control, which shares the same task-minibatch budget per update as collection matching; however, the mean success rates at the endpoint differ by only about 4 percentage points, and the control group exhibits significant variation across seeds (93.3% ± 4.9%, based on only 3 seeds). Figure 4 shows that, under the “naive joint” approach, the decay of ensemble disagreement and the intrinsic bonus coincides with lower success rates on difficult tasks; however, these results alone do not prove that the decay of the exploration signal is the primary causal mediator of the performance gap. The authors explicitly acknowledge this limitation in Section 4.3. Furthermore, Algorithm 1 selects data collection tasks based on episode boundaries, whereas the UTD analysis in Equation (6) uses the proportion of tasks in the environmental interaction data. If the actual episode lengths vary across different tasks, the probability of a task being selected may not necessarily equal the proportion of transitions it contributes. The paper has not yet explicitly provided empirical statistics sufficient to verify this consistency. Collection matching is one of the paper’s core contributions. If the key benefits stem primarily from single-task interleaved updates, changes in the overall optimization volume, or other coupled optimization effects—rather than strictly from matching data collection proportions—this would alter the assessment of this mechanism’s independent contribution and the strength of its theoretical explanation.

Questions for authors
  1. The MT50 results in Tables 1 and 3 are all from a single training run. Could the authors provide results from additional independent training seeds, or evidence from the existing experiments demonstrating stability under different initializations, or rerun at least MOORE/MoW/MOORE+HyMTRL under the same conditions? Additionally, does the current selection rule of single training run + best checkpoint depend on the evaluation set? Could the authors clarify the impact of the evaluation and selection protocol for the chosen checkpoint on the final reported results?
  2. Equation (6) assumes that the task sampling probability matches the optimization sampling probability, but Algorithm 1 collects data based on task selection at the end of each episode. Could the authors clarify whether the episode lengths for different tasks are fixed, and whether using task selection weights is indeed equivalent to matching the sampling ratio of environment transitions? Could they provide measured per-task collected transitions, update counts, and UTD to verify that these values approach the theoretical expectation of 1/k during training?
  3. To what extent is the endpoint advantage of “collection-matched” over “uniform random-single” in Table 2 consistent across the three seeds? Can the authors provide results for paired seeds or the differences between each training seed, and further explain why the AUC of “naive joint” is higher than that of “collection-matched,” even though its final success rate is significantly lower?
7 comments
AuthorsOpenReview ↗

Response to Weakness 1 — Robustness of the MT50 result

Thank you for raising the limits of the original MT50 evaluation. By the submission deadline we had completed one training run, and the reported standard deviation measured variation across evaluation goals for a single selected checkpoint. We have now evaluated five independently trained policies at the original 12M budget and at the extended 15M budget. The last column shows the highest evaluation score observed within each run.

Training seedSuccess at 12M (%)Success at 15M (%)Best observed through 15M (%)
4273.289.691.0
6676.486.688.6
10073.084.485.8
58580.687.288.4
202680.285.285.4
Mean ± SD across training seeds76.7 ± 3.786.6 ± 2.087.8 ± 2.3

At the fixed 15M endpoint, all five runs reached at least 84.4% success. Within these same five new runs, selecting the highest-scoring checkpoint separately for each run raised the mean by 1.24 percentage points. We use the fixed endpoint as our primary result.

The three baselines you highlighted differ in which parts of our MT50 protocol they share. The table gives their published settings and scores alongside our fixed-endpoint result.

MT50 methodObservationsEpisode horizon (steps)Environment interactionsSuccess (%)
MOOREState150100M72.9 ± 3.3
MoWImage50015M74.5 ± 1.1
MOORE + HyMTRLState20075M86.2 ± 3.4
WAM-MTRL-VEState15015M86.6 ± 2.0

MoW matches our interaction budget. MOORE matches our observations and horizon. Comparing the published scores therefore gives context for the five-seed result, while a controlled ranking would require a common evaluation and training protocol.

AuthorsOpenReview ↗

Response to Weakness 2 — The scheduling effect and the role of intrinsic rewards

Figure 5 of the revised paper shows declining exploration signals alongside limited success on difficult tasks under naive joint updates. To test whether the scheduling advantage depends on these signals entering the reward, we compared both schedules with intrinsic rewards enabled and disabled on the same five MT10 seeds. Disabling the bonuses sets both intrinsic-reward coefficients to zero while retaining world-model training and auxiliary losses. The table reports success at exactly 1M environment steps. We computed each scheduling gain within a seed before averaging.

Intrinsic rewardsCollection-matched success (%)Naive joint success (%)Paired scheduling gain (percentage points)
Off90.4 ± 3.169.8 ± 4.1+20.6 ± 4.3
On96.2 ± 1.672.2 ± 5.3+24.0 ± 4.7

With environment rewards alone, collection matching outperformed naive joint updates on all five seeds, with a mean paired gain of 20.6 ± 4.3 percentage points. The scheduling advantage therefore persists when both intrinsic bonuses are disabled. Enabling the bonuses increased mean final success under both schedules, but the change in the scheduling gap was 3.4 ± 6.8 percentage points across paired seeds. This variability leaves the interaction between scheduling and intrinsic rewards uncertain. The observed decline in exploration signals does not establish the cause of the performance gap.

Naive joint updates process one minibatch per ready task at each tick. To isolate the choice of task at a fixed update budget, we compared collection matching with uniform single-task sampling. Both process one task minibatch every two environment steps. The table gives the five paired endpoints and also shows the effect of disabling the bonuses under collection matching.

Training seedMatched, bonuses on (%)Uniform single-task, bonuses on (%)Matched, bonuses off (%)
42979292
66958585
100979493
585949091
2026989291
Mean ± SD96.2 ± 1.690.6 ± 3.490.4 ± 3.1

At the same task-minibatch budget and with bonuses enabled, collection matching outperformed uniform sampling on every seed, with a paired gain of 5.6 ± 2.7 percentage points. Within collection matching, enabling the bonuses improved success on every seed by 5.8 ± 2.8 points on average.

We also extended the sampling comparison to MT50 using seeds 42, 100 and 2026. Both variants use the full agent with six experts, the same collection rule and evaluation protocol, a 15M-step budget, and one task minibatch every two environment steps. Only the update-task sampling rule changes.

MT50 update samplingSuccess at 15M (%)Normalized AUC over 0–15M (%)
Collection-matched86.4 ± 2.855.76 ± 1.44
Uniform single-task67.1 ± 4.955.75 ± 2.49

Collection matching improved all three paired endpoints by 24.8, 20.6 and 12.4 percentage points, respectively, giving a mean gain of 19.3 ± 6.3 points. This comparison uses the corresponding three-seed subset of the full-agent runs; the five-seed benchmark result remains 86.6% ± 2.0%. The similar AUCs show that the benefit is concentrated in later learning and final performance, rather than an advantage throughout training.

The measured allocation is consistent with the intended scheduling effect. All 66,660 completed episodes in the ten MT10 diagnostic runs for collection matching and uniform sampling lasted 150 steps. From the time all buffers were ready through 1M steps, the five-seed mean update-to-data ratio (UTD) for each task fell between 0.489 and 0.515 under collection matching, close to the reference of 0.5. Under uniform sampling the range was 0.249 to 1.409. Additional diagnostics for naive joint and freshness-gated updates are reported in our response to Question 2. They also show why cumulative UTD alignment alone is insufficient to explain all differences in performance.

The learning curves explain the AUC ordering on the five shared MT10 seeds with bonuses enabled. Checkpoint values are mean success across all ten tasks and five seeds. AUC is computed from the unsmoothed evaluations within each run before averaging across seeds.

Schedule0.3M (%)0.7M (%)1M (%)Normalized AUC (%)
Naive joint51.069.672.255.24 ± 3.35
Collection-matched23.481.296.252.76 ± 1.75

The early lead gave naive joint updates a higher AUC. Collection matching had overtaken joint updates by 0.7M. At 1M it reached 90.0% mean success on Push, Pick-Place and Peg-Insert-Side, compared with 46.0% under joint updates. The task-level analysis in our response to Question 3 shows how early progress on the other seven tasks contributes to the AUC advantage of naive joint updates.

AuthorsOpenReview ↗

Response to Question 1 — Independent MT50 runs, checkpoint selection and episode length

We extended the MT50 budget from 12M to 15M environment interactions. Across the five independent runs, mean success was 76.7% ± 3.7% at 12M and 86.6% ± 2.0% at 15M. Our primary result uses the unsmoothed evaluation at exactly 15M steps. We evaluate the policy directly in the simulator and count an episode as successful if the simulator signals success at any step. Figure 8 of the revision shows the learning curve for each MT50 task.

Training seedSuccess at 15M (%)Best observed success through 15M (%)
4289.691.0
6686.688.6
10084.485.8
58587.288.4
202685.285.4
Mean ± sample SD86.6 ± 2.087.8 ± 2.3

In the original submission, we selected a checkpoint using success rates from simulator evaluations during training, then reevaluated that policy on five independently sampled sets of evaluation goals. The fixed-budget result uses the evaluation at the specified step. Selecting the highest observed score separately within each 15M run instead raises the mean by 1.24 percentage points.

In Table 1, the episode horizon HH is the maximum number of environment steps per episode. We use H=150H=150, as do MTSAC, CARE, PaCo and MOORE. MOORE + HyMTRL uses H=200H=200, and MoW uses H=500H=500. If both evaluations count an episode as successful as soon as the simulator signals success, extending the same trajectory beyond 150 steps preserves any success already observed. With the policy, initial state, goal and environment held fixed, the extra steps can give an episode that has yet to succeed another opportunity. Our 150-step cap therefore confers no advantage from extra time within an episode.

Table 1 gives the published baseline scores under each study's own protocol. Our entry is the five-seed mean at the fixed 15M endpoint. The revised table lists each method's episode horizon, observation modality and interaction budget. These protocol differences limit the conclusions we can draw about relative performance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.