- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
In this paper, the authors discuss the benefits of mixing gradients from different domains when post-training for agentic RL. To do so, the authors introduced mixed-task policy updates with asynchronous rollout generation as well as new sampling strategy called TBS, that balances exploration, reinforcement, and retention as new tasks and data are introduced during long time trainings. Empirical results show the benefit of the proposed method, Maco-RL, for improving, for instance, Qwen3.5-27B overall performance to be comparable to that of Qwen3.8-27B across five agentic benchmarks of three domains.
Critical strengths
- The paper is not hard to follow, and the plots are clear.
- The empirical results are dense and show promising results for the agentic field.
- The Tri-Buffer Sampling seems novel, yet it is a very engineered component.
- The ablation of Gradient Mixing Improves Training Stability sounds interesting. The entropy behavior under alternative or mixed gradients is a nice addition.
Critical weaknesses
- The crucial weakness in this work is its lack of novelty. This work tries to introduce a very old learning paradigm differently in the agentic RL domain. For some reason, the authors used the term "Mixed-Task Reinforcement Learning" to refer to Multi-Task Reinforcement Learning, a very well-known and established learning paradigm that relies on learning a shared policy with mixed gradients coming from different task objectives. In [2], the benefit of learning a shared policy trained with different task objectives has been theoretically proven. Hence, the contribution of this work is very limited, if any, while the connection to a large body of literature is missing.
- Regarding the contributions stated, the only component that might be new is the TBS, which is more engineering than science. In addition, there is another large body of work within multi-task reinforcement learning that illustrates different task sampling strategies [3] or prioritizing transitions or samples [4], which the authors have never mentioned or compared in this paper.
- Evaluations seem to be conducted and presented with a single seed.
In my honest opinion, this work is reintroducing known concepts from the multi-task reinforcement learning literature, intentionally or unintentionally, to solve a new application. This work is proposing more of an engineering-based or system-based method than science, which seems to show potential results as stated in the paper, yet it is not an addition to the research field.
[1] Vithayathil Varghese, Nelson, and Qusay H. Mahmoud. "A survey of multi-task deep reinforcement learning." Electronics 9.9 (2020): 1363.
[2] D'Eramo, Carlo, et al. "Sharing knowledge in multi-task deep reinforcement learning." arXiv preprint arXiv:2401.09561 (2024).
[3] Cho, Myungsik, et al. "Hard Tasks First: Multi-Task Reinforcement Learning Through Task Scheduling." ICML. 2024.
[4] Schaul, Tom, et al. "Prioritized experience replay." arXiv preprint arXiv:1511.05952 (2015).
Questions for authors
- What is the real contribution of this paper relative to the literature on multi-task reinforcement learning? And why was mixed-task reinforcement learning introduced as a new learning paradigm? How does it differ from multi-task reinforcement learning?
- How have you decided on the percentages (contributions) of each of the replay buffers?
- How many seeds were used to report the results in the paper?
- What is exactly? I believe it is only shown in Figure 6 and used directly in the Figure 7 discussion section.
- What does "Maco" stand for?
5 comments
Response to Reviewer tweZ (2/2)
Q2: TBS is more engineering than science. In addition, there is another large body of work within multi-task reinforcement learning that illustrates different task sampling strategies [1] or prioritizing transitions or samples [2], which the authors have never mentioned or compared in the paper.
A2: We thank the reviewer for highlighting the related sampling literature. TBS provides a unified sampling framework for long-horizon LLM-agent RL. Through a simple quota-based strategy, it addresses multiple key challenges in agentic RL: allocating limited rollout compute among reinforcing emerging capabilities, exploring unsolved tasks, and retaining earlier knowledge, while supporting continued training as new data and domains are introduced.
Our ablation supports the effectiveness of TBS, with higher FeatureBench scores than uniform sampling at all five evaluated checkpoints:
| Training step | Vanilla Tests Passed (%) | TBS Tests Passed (%) | Gain (pp) |
|---|---|---|---|
| 5 | 32.62 | 39.89 | +7.27 |
| 10 | 35.43 | 39.01 | +3.58 |
| 15 | 36.77 | 41.85 | +5.08 |
| 20 | 37.88 | 39.57 | +1.69 |
| 25 | 39.78 | 42.87 | +3.09 |
We thank the reviewer for pointing out the cited sampling methods and will discuss their relationship to TBS in the revised manuscript.
[1] Cho, Myungsik, et al. "Hard Tasks First: Multi-Task Reinforcement Learning Through Task Scheduling." ICML. 2024.
[2] Schaul, Tom, et al. "Prioritized experience replay." arXiv preprint arXiv:1511.05952 (2015).
Q3: How have you decided on the percentages (contributions) of each of the replay buffers?
A3: We set Reinforcement : Frontier : History = 3 : 1 : 1, corresponding to 60%, 20%, and 20% of the sampling quota.
The motivation is to spend most rollouts on instances with observed but unreliable success, where reward variation can give useful gradient signals, while reserving smaller, explicit budgets for exploration and exposure to earlier-stage tasks.
Q4: How many seeds were used to report the results in the paper?
A4: We used a single fixed random seed for all benchmark evaluations to make the results as reproducible as possible.
Q5: What is exactly? I believe it is only shown in Figure 6 and used directly in the Figure 7 discussion section.
A5: denotes the n-th policy version synchronized to the rollout workers, with the version counter advancing at each weight synchronization. We thank the reviewer for pointing out the missing definition. We will add this clarification to the caption of Figure 6 and the discussion of Figure 7.
Q6: What does "Maco" stand for?
A6: “Maco” stands for “more than another coder.” The name reflects our goal of using mixed-task RL to extend agent capabilities beyond writing code to a broader range of tasks, including office automation, terminal operations, and other real-world workflows. We will add this explanation in a footnote on the first page.
Looking forward to your valuable feedback!
Dear Reviewer,
We hope this message finds you well. With less than three days remaining in the discussion period, we wanted to follow up and hope our response has addressed your concerns. If there are any additional questions or suggestions, please let us know. Your feedback has been valuable in helping us improve our work, and we welcome any further thoughts you may have.
Thank you again for your time and thoughtful feedback!
Response to Reviewer tweZ (1/2)
We thank the reviewer for the close reading and for highlighting the promising results and the gradient-mixing stability analysis. Below, we address the concerns about novelty and prior work by clarifying how Maco-RL builds on multi-task RL and what its design and experiments contribute.
Q1: What is the real contribution of this paper relative to the literature on multi-task reinforcement learning? And why was mixed-task reinforcement learning introduced as a new learning paradigm? How does it differ from multi-task reinforcement learning?
A1: We appreciate the reviewer’s thoughtful feedback. Our work addresses an important open problem in LLM post-training: how to make direct mixed-task RL stable and scalable across long-horizon LLM-agent tasks from multiple domains. Solving this problem offers a potentially more efficient and simpler alternative to the common practice of training separate domain specialists and consolidating them through OPD [1, 2], yet no prior work had publicly presented a workable approach or training recipe in this setting due to the challenges posed by task heterogeneity and scalability constraints. Recently, MiMo made its mixed-task RL livestream public, sparking broad community discussion (https://mimo.xiaomi.com/rl/). Our work was developed concurrently and complements MiMo’s effort by offering the community fully open training data and configurations, controlled ablations, and practical insights into training failure modes.
Training separate specialists followed by OPD requires multiple RL runs and an additional consolidation stage. Direct mixed-task RL offers the potential to simplify this pipeline while enabling cross-task transfer within a shared policy. Our work extends the scope of established MTRL to long-horizon, multi-domain LLM-agent training, offering a new perspective and training paradigm for long-horizon agentic RL in LLM post-training.
However, realizing this extension is challenging. The cited studies, including [3], primarily focus on conventional multi-task deep RL, leaving unresolved the challenges of stable, scalable joint training across LLM-agent tasks with diverse agent frameworks, large GPU idle time, and severe off-policy issues. Moreover, scaling to new tasks and domains poses an additional challenge, as costly long-horizon rollouts make it difficult to balance exploration and retention within a limited compute budget.
We address these challenges through quota-based asynchronous mixed-task training and unified sampling for exploration and retention as tasks expand, while validating our approach across models and training stages. Our analyses of training failures were also recognized as “very useful for the community” by Reviewer wYqT and “unusually informative” by Reviewer 7ndF. Specifically, our paper makes three main contributions:
Overall, we provide an evaluated approach and empirical insights for applying mixed-task RL to heterogeneous, long-horizon LLM-agent training. We hope this clarifies our contributions, and we will revise the manuscript accordingly and add the relevant references.
[1] DeepSeek-AI. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. 2026. arXiv:2606.19348.
[2] Kimi Team. Kimi K3: Open Frontier Intelligence. 2026. arXiv:2607.24653.
[3] D'Eramo, Carlo, et al. "Sharing knowledge in multi-task deep reinforcement learning." arXiv preprint arXiv:2401.09561 (2024).