acceptodds
Under review as a conference paper at ICLR 2027

Maco-RL: Scaling Agent Capabilities with Mixed-Task Reinforcement Learning

Abstract

Mixed-task reinforcement learning (MixRL) directly trains a unified policy across domains to continually expand agent capabilities without training separate specialists or requiring an extra distillation stage. Despite its appeal, scaling MixRL poses significant challenges in achieving stable and efficient mixed-task learning, effectively allocating rollout compute, and supporting continual domain expansion. To address these challenges, we propose Maco-RL, an approach that effectively scales MixRL. Specifically, Maco-RL employs a quota-based scheduler for asynchronous rollouts across multiple task domains, thereby stabilizing training by balancing gradient contributions and mitigating off-policy effects. Along an orthogonal axis, Maco-RL further introduces Tri-Buffer Sampling (TBS) to allocate compute among samples with varying levels of mastery, promoting sustained frontier exploration while preserving acquired capabilities. Experiments on five benchmarks show Maco-RL raises Qwen3.5-27B's overall score from 43.6 to 54.2 and then to 60.1 with continual expansion, reaching the frontier held by Qwen3.8-27B, which is further pushed outward by +5.7. Maco-RL also outperforms single-task RL with less compute for policy updates. These results support MixRL as a practical and compute-efficient post-training approach with substantial potential to strengthen general agent capabilities.

Primary area:
reinforcement learning
Keywords:
Agentic Reinforcement Learning, Multi-Environment Training, Adaptive Task Sampling, Large Language Model Agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_tweZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

In this paper, the authors discuss the benefits of mixing gradients from different domains when post-training for agentic RL. To do so, the authors introduced mixed-task policy updates with asynchronous rollout generation as well as new sampling strategy called TBS, that balances exploration, reinforcement, and retention as new tasks and data are introduced during long time trainings. Empirical results show the benefit of the proposed method, Maco-RL, for improving, for instance, Qwen3.5-27B overall performance to be comparable to that of Qwen3.8-27B across five agentic benchmarks of three domains.

Critical strengths
  • The paper is not hard to follow, and the plots are clear.
  • The empirical results are dense and show promising results for the agentic field.
  • The Tri-Buffer Sampling seems novel, yet it is a very engineered component.
  • The ablation of Gradient Mixing Improves Training Stability sounds interesting. The entropy behavior under alternative or mixed gradients is a nice addition.
Critical weaknesses
  • The crucial weakness in this work is its lack of novelty. This work tries to introduce a very old learning paradigm differently in the agentic RL domain. For some reason, the authors used the term "Mixed-Task Reinforcement Learning" to refer to Multi-Task Reinforcement Learning, a very well-known and established learning paradigm that relies on learning a shared policy with mixed gradients coming from different task objectives. In [2], the benefit of learning a shared policy trained with different task objectives has been theoretically proven. Hence, the contribution of this work is very limited, if any, while the connection to a large body of literature is missing.
  • Regarding the contributions stated, the only component that might be new is the TBS, which is more engineering than science. In addition, there is another large body of work within multi-task reinforcement learning that illustrates different task sampling strategies [3] or prioritizing transitions or samples [4], which the authors have never mentioned or compared in this paper.
  • Evaluations seem to be conducted and presented with a single seed.

In my honest opinion, this work is reintroducing known concepts from the multi-task reinforcement learning literature, intentionally or unintentionally, to solve a new application. This work is proposing more of an engineering-based or system-based method than science, which seems to show potential results as stated in the paper, yet it is not an addition to the research field.

[1] Vithayathil Varghese, Nelson, and Qusay H. Mahmoud. "A survey of multi-task deep reinforcement learning." Electronics 9.9 (2020): 1363.

[2] D'Eramo, Carlo, et al. "Sharing knowledge in multi-task deep reinforcement learning." arXiv preprint arXiv:2401.09561 (2024).

[3] Cho, Myungsik, et al. "Hard Tasks First: Multi-Task Reinforcement Learning Through Task Scheduling." ICML. 2024.

[4] Schaul, Tom, et al. "Prioritized experience replay." arXiv preprint arXiv:1511.05952 (2015).

Questions for authors
  • What is the real contribution of this paper relative to the literature on multi-task reinforcement learning? And why was mixed-task reinforcement learning introduced as a new learning paradigm? How does it differ from multi-task reinforcement learning?
  • How have you decided on the percentages (contributions) of each of the replay buffers?
  • How many seeds were used to report the results in the paper?
  • What is VnV_n exactly? I believe it is only shown in Figure 6 and used directly in the Figure 7 discussion section.
  • What does "Maco" stand for?
5 comments
AuthorsOpenReview ↗

Response to Reviewer tweZ (1/2)

We thank the reviewer for the close reading and for highlighting the promising results and the gradient-mixing stability analysis. Below, we address the concerns about novelty and prior work by clarifying how Maco-RL builds on multi-task RL and what its design and experiments contribute.


Q1: What is the real contribution of this paper relative to the literature on multi-task reinforcement learning? And why was mixed-task reinforcement learning introduced as a new learning paradigm? How does it differ from multi-task reinforcement learning?

A1: We appreciate the reviewer’s thoughtful feedback. Our work addresses an important open problem in LLM post-training: how to make direct mixed-task RL stable and scalable across long-horizon LLM-agent tasks from multiple domains. Solving this problem offers a potentially more efficient and simpler alternative to the common practice of training separate domain specialists and consolidating them through OPD [1, 2], yet no prior work had publicly presented a workable approach or training recipe in this setting due to the challenges posed by task heterogeneity and scalability constraints. Recently, MiMo made its mixed-task RL livestream public, sparking broad community discussion (https://mimo.xiaomi.com/rl/). Our work was developed concurrently and complements MiMo’s effort by offering the community fully open training data and configurations, controlled ablations, and practical insights into training failure modes.

Training separate specialists followed by OPD requires multiple RL runs and an additional consolidation stage. Direct mixed-task RL offers the potential to simplify this pipeline while enabling cross-task transfer within a shared policy. Our work extends the scope of established MTRL to long-horizon, multi-domain LLM-agent training, offering a new perspective and training paradigm for long-horizon agentic RL in LLM post-training.

However, realizing this extension is challenging. The cited studies, including [3], primarily focus on conventional multi-task deep RL, leaving unresolved the challenges of stable, scalable joint training across LLM-agent tasks with diverse agent frameworks, large GPU idle time, and severe off-policy issues. Moreover, scaling to new tasks and domains poses an additional challenge, as costly long-horizon rollouts make it difficult to balance exploration and retention within a limited compute budget.

We address these challenges through quota-based asynchronous mixed-task training and unified sampling for exploration and retention as tasks expand, while validating our approach across models and training stages. Our analyses of training failures were also recognized as “very useful for the community” by Reviewer wYqT and “unusually informative” by Reviewer 7ndF. Specifically, our paper makes three main contributions:

  1. Quota-based asynchronous mixed-task training. Maco-RL uses per-environment concurrency control to accommodate uneven rollout costs and fixed task quotas to balance domains in every update. In our asynchronous setting, we find that alternating updates lead to tool-use degradation and reward collapse, whereas mixed-task updates remain stable, highlighting the importance of update composition under policy lag.
  2. Unified sampling for exploration and retention as tasks expand. TBS assigns separate rollout quotas to occasionally solved, unsolved, and earlier-stage instances to balance reinforcement, exploration, and retention. This unified strategy supports continued training as new tasks and domains are introduced.
  3. Cross-model and cross-stage validation. On Qwen3.5-27B, the task pool expands from 11,086 instances across three task types to 21,538 across four, while Overall improves from 43.6 to 54.2 and then 60.1. Applying the same recipe to Qwen3.8-27B further improves Overall from 61.4 to 67.0, making it SOTA at this model size.

Overall, we provide an evaluated approach and empirical insights for applying mixed-task RL to heterogeneous, long-horizon LLM-agent training. We hope this clarifies our contributions, and we will revise the manuscript accordingly and add the relevant references.

[1] DeepSeek-AI. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. 2026. arXiv:2606.19348.
[2] Kimi Team. Kimi K3: Open Frontier Intelligence. 2026. arXiv:2607.24653.
[3] D'Eramo, Carlo, et al. "Sharing knowledge in multi-task deep reinforcement learning." arXiv preprint arXiv:2401.09561 (2024).

AuthorsOpenReview ↗

Response to Reviewer tweZ (2/2)

Q2: TBS is more engineering than science. In addition, there is another large body of work within multi-task reinforcement learning that illustrates different task sampling strategies [1] or prioritizing transitions or samples [2], which the authors have never mentioned or compared in the paper.

A2: We thank the reviewer for highlighting the related sampling literature. TBS provides a unified sampling framework for long-horizon LLM-agent RL. Through a simple quota-based strategy, it addresses multiple key challenges in agentic RL: allocating limited rollout compute among reinforcing emerging capabilities, exploring unsolved tasks, and retaining earlier knowledge, while supporting continued training as new data and domains are introduced.

Our ablation supports the effectiveness of TBS, with higher FeatureBench scores than uniform sampling at all five evaluated checkpoints:

Training stepVanilla Tests Passed (%)TBS Tests Passed (%)Gain (pp)
532.6239.89+7.27
1035.4339.01+3.58
1536.7741.85+5.08
2037.8839.57+1.69
2539.7842.87+3.09

We thank the reviewer for pointing out the cited sampling methods and will discuss their relationship to TBS in the revised manuscript.

[1] Cho, Myungsik, et al. "Hard Tasks First: Multi-Task Reinforcement Learning Through Task Scheduling." ICML. 2024.
[2] Schaul, Tom, et al. "Prioritized experience replay." arXiv preprint arXiv:1511.05952 (2015).


Q3: How have you decided on the percentages (contributions) of each of the replay buffers?

A3: We set Reinforcement : Frontier : History = 3 : 1 : 1, corresponding to 60%, 20%, and 20% of the sampling quota.

The motivation is to spend most rollouts on instances with observed but unreliable success, where reward variation can give useful gradient signals, while reserving smaller, explicit budgets for exploration and exposure to earlier-stage tasks.


Q4: How many seeds were used to report the results in the paper?

A4: We used a single fixed random seed for all benchmark evaluations to make the results as reproducible as possible.


Q5: What is VnV_n exactly? I believe it is only shown in Figure 6 and used directly in the Figure 7 discussion section.

A5: VnV_n denotes the n-th policy version synchronized to the rollout workers, with the version counter advancing at each weight synchronization. We thank the reviewer for pointing out the missing definition. We will add this clarification to the caption of Figure 6 and the discussion of Figure 7.


Q6: What does "Maco" stand for?

A6: “Maco” stands for “more than another coder.” The name reflects our goal of using mixed-task RL to extend agent capabilities beyond writing code to a broader range of tasks, including office automation, terminal operations, and other real-world workflows. We will add this explanation in a footnote on the first page.

AuthorsOpenReview ↗

Looking forward to your valuable feedback!

Dear Reviewer,

We hope this message finds you well. With less than three days remaining in the discussion period, we wanted to follow up and hope our response has addressed your concerns. If there are any additional questions or suggestions, please let us know. Your feedback has been valuable in helping us improve our work, and we welcome any further thoughts you may have.

Thank you again for your time and thoughtful feedback!

Reviewer_wYqTOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Maco-RL, a framework for scaling direct mixed-task reinforcement learning (MixRL) for LLM agents across software engineering, terminal operation, office automation, and feature development tasks. To handle heterogeneous task rollout times and continual data/domain expansion without training separate domain specialists, Maco-RL introduces: (1) an asynchronous rollout–training architecture with per-environment concurrency slots and within-batch gradient mixing across all task types to mitigate off-policy drift; and (2) Tri-Buffer Sampling (TBS), which divides task pools into Frontier, Reinforcement, and History buffers to balance exploration of unsolved instances, Gaussian-weighted consolidation of partially solved instances, and replay of prior-stage tasks. Evaluated on Qwen3.5-27B (across two incremental stages) and Qwen3.8-27B (in a single stage) over five agentic benchmarks, Maco-RL achieves substantial overall score improvements and demonstrates strong training stability and data efficiency compared to single-task RL and alternating updates.

Critical strengths
  1. The asynchronous mixed-task RL system is well-engineered, and the empirical analysis of training stability is very convincing. In particular, showing how alternating single-task updates interact with asynchronous policy lag to cause tool-use failures and reward collapse, while within-batch gradient mixing stabilizes training (Figures 6 and 7), is a practical insight for scaling multi-turn agentic RL. The candid analysis of reward hacking (such as git history leaks) and LLM-judge protocol collapse is also very useful for the community.

  2. The overall empirical gains are strong across two model generations. Maco-RL substantially improves the overall performance of both Qwen3.5-27B and Qwen3.8-27B across five agentic benchmarks, and demonstrates better sample efficiency and positive cross-task transfer compared to training separate single-task RL models.

Critical weaknesses
  1. On Qwen3.5-27B, the reported +16.6 overall improvement in Table 1 is heavily skewed toward a single benchmark, Claw-Anything (+52.6 points, from 12.0 to 64.6), which shares the same generation pipeline as its training environment and notably does not even use TBS (it uses uniform sampling). In contrast, on the coding benchmarks where TBS is actually applied, the gains on Qwen3.5-27B are quite small: SWE-bench Verified slightly drops from 67.4 to 67.0 in Stage 1 and only reaches 68.6 (+1.2) in Stage 2, while FeatureBench Tests Passed only improves from 37.9 to 39.9 (+2.0). Furthermore, although two-stage continual expansion and the History Buffer are central claims of the paper, the stronger Qwen3.8-27B model is trained in a single stage with all data available upfront (so the History Buffer is empty) and with thinking mode enabled during rollouts (unlike Qwen3.5-27B where thinking mode is disabled).

  2. The ablation of Tri-Buffer Sampling (TBS) and the compute comparison against single-task baselines are somewhat limited. In Figure 8, TBS is only compared against uniform Vanilla sampling over 25 steps on FeatureBench, without isolating the effect of the History Buffer during the Stage 1 to Stage 2 transition or comparing against closely related competence-based Gaussian sampling methods. In addition, Figure 1(b) compares mixed-task RL and single-task RL using total optimizer updates rather than actual rollout tokens or GPU hours, which may not be a fair compute proxy since rollout durations vary widely across tasks. Despite motivating MixRL as an alternative to specialist RL followed by on-policy distillation (OPD), no empirical comparison with a specialist + OPD pipeline is included.

Questions for authors
  1. Why does Maco-RL-3.5 show only minimal gains on SWE-bench Verified (+1.2 in S2 and -0.4 in S1) despite OpenSWE taking 50% of the rollout budget in S1, whereas Maco-RL-3.8 gains +8.8 points? How much of the coding gain on Qwen3.8-27B comes from enabling thinking mode during training rather than MixRL itself?

  2. Did you run an ablation in Stage 2 without the 20% History Buffer quota to directly measure how much it helps prevent catastrophic forgetting on Stage 1 tasks?

  3. How does the efficiency comparison between Maco-RL and single-task RL in Figure 1(b) look if plotted against total generated rollout tokens or wall-clock GPU hours instead of optimizer steps?

4 comments
AuthorsOpenReview ↗

Response to Reviewer wYqT (1/2)

We are glad that the reviewer found the training-stability analysis convincing and the practical lessons from environment failures useful. Accordingly, each question is addressed below.


Q1: On Qwen3.5-27B, the reported +16.6 overall improvement in Table 1 is heavily skewed toward a single benchmark, Claw-Anything (+52.6 points, from 12.0 to 64.6), which shares the same generation pipeline as its training environment and notably does not even use TBS (it uses uniform sampling).

A1: We thank the reviewer for this careful observation, which helps us clarify the interpretation of our results and improve the paper.

Qwen3.5-27B had not previously been trained on Claw-style tasks, whereas Qwen3.8-27B had already received such training. Their initial Claw-Anything Pass@1 scores are 12.0% and 46.0%, respectively. Qwen3.5 therefore starts from a substantially lower baseline, leaving greater room for improvement and helping explain its larger gains on this benchmark.

However, the improvements are not limited to Claw-Anything. Excluding this benchmark and averaging the remaining four benchmark metrics in Table 1, both models still achieve substantial gains:

ModelBase modelAfter Maco-RLGain
Qwen3.5-27B → Maco-RL-3.5 (S2)51.559.0+7.5
Qwen3.8-27B → Maco-RL-3.865.270.0+4.8

These averages use Claw-Eval Pass$^3$, Terminal-Bench 2.1 Pass@1, SWE-bench Verified Pass@1, and FeatureBench Tests Passed; gains are in percentage points, calculated from the displayed scores. Maco-RL-3.8 also has the highest Overall score among the evaluated 27B models.

Uniform sampling for Claw is deliberate. TBS is designed for sparse-reward settings where many rollout groups receive all-zero rewards, yielding no reward-based RLOO advantage. In contrast, Claw provides continuous, partial-credit rewards below the success threshold (Appendix A.3), making useful gradient signals readily available. We therefore use vanilla uniform sampling for Claw. The FeatureBench ablation demonstrates the effectiveness of TBS in the challenging, sparse-reward feature-development setting:

Sampling strategyFeatureBench Tests Passed (%)
Uniform sampling39.8
TBS42.9

After 25 training steps, TBS outperforms vanilla sampling by 3.1 percentage points, supporting its effectiveness in this sparse-reward setting.


Q2: Although two-stage continual expansion and the History Buffer are central claims of the paper, the stronger Qwen3.8-27B model is trained in a single stage with all data available upfront (so the History Buffer is empty).

A2: We thank the reviewer for the careful reading. Our task pool expands continually. When we began training Qwen3.8-27B after completing the two-stage Qwen3.5-27B experiment, data from all four domains were ready for training. We therefore did not deliberately divide Qwen3.8 training into multiple stages. We would also like to clarify that the two experiments serve different purposes. The Qwen3.5-27B experiment evaluates continued training as the task pool expands and feature development is introduced as a new environment type in S2, while the Qwen3.8-27B experiment tests how much the complete mixed-task RL recipe improves a stronger base model, with all four task types available. We will state these experimental purposes more clearly in the revised manuscript.


Q3: Why does Maco-RL-3.5 show only minimal gains on SWE-bench Verified (+1.2 in S2 and -0.4 in S1) despite OpenSWE taking 50% of the rollout budget in S1, whereas Maco-RL-3.8 gains +8.8 points? How much of the coding gain on Qwen3.8-27B comes from enabling thinking mode during training rather than MixRL itself?

A3: During training, overall performance and individual benchmark performance do not always improve at the same time. When considering only SWE-bench performance, we could actually observe larger gains at other points:

Model / stageStage-start scoreBest SWE-bench scoreGain
Qwen3.5-27B, S167.369.0+1.7
Qwen3.5-27B, S267.072.6+5.6
Qwen3.8-27B70.279.8+9.6

As shown in the table, Qwen3.5 improves by 5.6 points in S2, a substantial gain more comparable to Qwen3.8’s 9.6-point improvement. Moreover, its gain increases from 1.7 points in S1 to 5.6 points in S2 after feature-development tasks are introduced, suggesting that MixRL itself can benefit SWE performance through cross-task transfer from these tasks. This may also help explain Qwen3.8’s larger gain, as its training includes these tasks from the start.

AuthorsOpenReview ↗

Response to Reviewer wYqT (2/2)

Q4: Did you run an ablation in Stage 2 without the 20% History Buffer quota to directly measure how much it helps prevent catastrophic forgetting on Stage 1 tasks?

A4: We thank the reviewer for this suggestion. We have started an ablation that trains Qwen3.8 on newly added TMax and OpenSWE data, with and without the History Buffer. It's expected that removing the History Buffer will reduce performance on Claw and feature-development tasks. The experiment requires over 100 GPUs for nearly a week to obtain reliable results. We will try our best to share the results by November 16 and will report any valuable findings as soon as they become available.


Q5: In Figure 8, TBS is only compared against uniform Vanilla sampling over 25 steps on FeatureBench, without [...] comparing against closely related competence-based Gaussian sampling methods.

A5: TBS is designed to address the lack of learning signals on difficult tasks: when all rollouts in a group fail, RLOO/GRPO-style algorithms yield zero advantages and generate no policy-gradient signal. Feature-development tasks are particularly difficult: 82.66% had no successful rollout initially, resulting in sparse training signals. Gaussian-only sampling (μ=0.5, σ=0.25) would still assign 55.18% of sampling probability to these unresolved tasks. TBS aims to make the balance between exploration and reinforcement independent of the data distribution. It sets explicit quotas for exploring difficult tasks and reinforcing existing capabilities, providing a trade-off between training efficiency and exploration.

We had also started an additional experiment with Gaussian-only sampling. Given the training cost, obtaining reliable results takes time, and we will make every effort to report reliable results by November 16 and incorporate them into the revised manuscript.


Q6: How does the efficiency comparison between Maco-RL and single-task RL in Figure 1(b) look if plotted against total generated rollout tokens or wall-clock GPU hours instead of optimizer steps?

A6: In mixed-task training, each policy update allocates the training batch equally across the three environments, with other settings matched to single-task RL. Therefore, three mixed-task optimizer updates use nearly the same total GPU hours and rollout tokens as the combined costs of updating each of the three specialists once. The measured GPU hours and rollout tokens are summarized below, showing only modest differences between the two approaches:

Training approachAccounted tokensRecorded GPU hoursBest-so-far mean Pass@1 (%)
Three specialists, combined1.821B13,090.664.28
Maco-RL1.852B12,572.468.41

As the table shows, Maco-RL achieves better performance than the specialists with similar GPU hours and rollout tokens. We will add these results to the revised manuscript.


Q7: Despite motivating MixRL as an alternative to specialist RL followed by on-policy distillation (OPD), no empirical comparison with a specialist + OPD pipeline is included.

A7: Figure 1(b) shows that mixed-task training is more efficient than training domain specialists. Adding OPD after training domain specialists incurs further cost without guaranteeing better performance than the specialists themselves. We are trying our best to build our OPD pipeline, which requires high-quality data for consolidating specialists and careful experimentation to develop a workable recipe. To obtain reliable results, we will continue this work beyond the discussion period and commit to including the comparison in the final version of the paper.

AuthorsOpenReview ↗

Update on the History Buffer Ablation

We thank the reviewer for suggesting this ablation. We now have results from the History Buffer ablation mentioned in our earlier response and would like to share them below. Starting from our trained Maco-RL-3.8 model, we continued training on newly added TMax and OpenSWE data, with and without the History Buffer. The table below reports performance on the two earlier domains after 40 additional training steps:

SettingClaw-Anything Pass@1 (%)Δ vs. initial (pp)FeatureBench Tests Passed (%)Δ vs. initial (pp)
Initial model55.21—54.20—
Without History Buffer53.47−1.7449.88−4.32
With History Buffer57.29+2.0854.37+0.17

Without the History Buffer, Claw-Anything and FeatureBench decline by 1.74 and 4.32 percentage points from the initial model. With the buffer, both scores are above their initial levels and exceed the no-history run by 3.82 and 4.49 points, respectively. These results support the History Buffer’s effectiveness in preserving previously learned capabilities during continued training. We will include these results in the revised manuscript.

AuthorsOpenReview ↗

Update on the Gaussian Sampling Comparison

We would like to share an update on the Gaussian sampling comparison discussed in our response to Q5. The experiment requires approximately two days of training on 96 GPUs. Given this high computational cost, additional time beyond the discussion period is needed to obtain reliable results. We commit to completing the experiment and including the results in the final version of the paper.

Comments on the paper

6 comments
AuthorsOpenReview ↗

Response to Reviewer 7ndF (1/2)

We appreciate the reviewer’s thoughtful feedback and recognition of the practical value of our work. The reviewer’s questions are addressed below.


Q1: Can you provide a comparison of mixed-task and specialist training using total generated tokens and GPU hours, including rollout generation, initial task screening, and discarded or restarted work? Existing logs would be sufficient to clarify the scope of the efficiency claim.

A1: We thank the reviewer for this meaningful suggestion; we have summarized the costs below:

Training approachAccounted tokensRecorded GPU hoursBest-so-far mean Pass@1 (%)
Three specialists, combined1.821B13,090.664.28
Maco-RL1.852B12,572.468.41

Including training, screening, and restarted work, Maco-RL achieves better performance with comparable token and GPU-hour costs. We will include this comparison and the accounting details in the revised manuscript.


Q2: The TBS comparison is limited to feature development (42.9 versus 39.8 Tests Passed at step 25), with no uncertainty reported. What is the paired task-level confidence interval for the FeatureBench difference?

A2: TBS is designed for difficult tasks in which all-zero reward groups yield no policy-gradient signal under RLOO/GRPO-style training. It reinforces emerging capabilities while reserving sampling capacity for unsolved instances. Because feature development is the most challenging of our four task types for the current model, FeatureBench provides a particularly suitable testbed for evaluating TBS.

Across the 100 FeatureBench tasks, the paired task-level bootstrap 95% confidence interval is [−2.21, +8.51] percentage points, with an observed improvement of +3.1 pp at step 25. Moreover, TBS achieves a higher score at every evaluated checkpoint:

Training stepTBS Tests Passed (%)Vanilla Tests Passed (%)Difference (pp)
539.8932.62+7.27
1039.0135.43+3.58
1541.8536.77+5.08
2039.5737.88+1.69
2542.8739.78+3.09

These results support the benefits of TBS over vanilla sampling on difficult tasks, as it explicitly balances training efficiency and frontier exploration.


Q3: In the existing TBS/Vanilla experiment, how many previously unsolved instances does Vanilla sampling solve over the same budget?

A3: TBS explicitly balances training efficiency and exploration, focusing on reinforcing existing capabilities while reserving capacity for exploration. After the same training steps, Vanilla solves 33 previously unsolved tasks, compared with 15 for TBS, yet achieves lower FeatureBench performance (39.78% versus 42.87%). This is consistent with their different sampling priorities: initially, 82.66% of feature-development tasks had no successful rollout, whereas TBS reserves a 20% exploration quota and allocates more capacity to reinforcement. Vanilla therefore discovers more new solutions due to its greater exposure to difficult tasks, but risks excessive exploration and lower learning efficiency, whereas TBS achieves a better balance and higher performance. We will include these results and analysis in the revised manuscript.


Q4: Can you provide a focused History Buffer removal comparison measuring performance on the S1 task types after S2?

A4: We appreciate the reviewer’s suggestion. We have begun training Qwen3.8 on newly added TMax and OpenSWE data, comparing runs with and without the buffer. The experiment is expected to show a substantial drop in performance on Claw and feature-development tasks when the History Buffer is removed. The experiment requires over 100 GPUs for nearly a week. We will try our best to share reliable results by November 16 and will report any valuable findings as soon as they become available.

AuthorsOpenReview ↗

Response to Reviewer 7ndF (2/2)

Q5: How were the checkpoints in Table 1 selected? Were the public benchmark instances also used to select checkpoints or tune the training procedure? Please distinguish validation from final evaluation and, where possible, report paired task-level uncertainty for the principal gains.

A5: We monitored training with five widely used public benchmarks and reported the results from the step with the highest Overall score. These benchmarks are well separated from the public training datasets [1, 2]. Benchmark scores were used to monitor progress, not to tune the training procedure. Each model was trained once with fixed hyperparameters commonly used by the community [3, 4].

As suggested by the reviewer, we add paired task-level confidence intervals for the principal gains:

MetricQwen3.5 → Maco-RL-3.5 S2; 95% CI for gainQwen3.8 → Maco-RL-3.8; 95% CI for gain
Claw-Eval Pass$^3$60.25 → 73.29; [+6.83, +19.88]71.43 → 74.53; [−2.48, +8.70]
Claw-Anything Pass@112.00 → 71.33; [+51.33, +67.33]46.00 → 52.67; [−1.33, +14.67]
Terminal-Bench 2.1 Pass@140.45 → 54.31; [+5.24, +22.47]68.91 → 72.28; [−1.50, +7.87]
SWE-bench Verified Pass@167.40 → 68.60; [−1.60, +4.00]70.20 → 79.00; [+5.80, +11.80]
FeatureBench Tests Passed37.93 → 39.94; [−3.10, +7.28]50.30 → 54.20; [−1.90, +9.92]
Overall43.61 → 61.49; [+14.99, +20.79]61.37 → 66.54; [+2.67, +7.68]

Both models show positive Overall gains (+17.88 and +5.17 percentage points), with 95% CIs excluding zero. For paired comparisons, Claw-Anything is evaluated on the common 150-task subset. We will add these results to the revised manuscript.

[1] Dayuan Fu et al. daVinci-Env: Open SWE Environment Synthesis at Scale. 2026. arXiv:2603.13023.
[2] Hamish Ivison et al. Tmax: A simple recipe for terminal agents. 2026. arXiv:2606.23321.
[3] Qiying Yu et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. 2025. arXiv:2503.14476.
[4] Michael Luo et al. DeepSWE: Training a Fully Open-sourced, State-of-the-Art Coding Agent by Scaling RL. 2025. Together AI Blog. https://www.together.ai/blog/deepswe

Dear Reviewers

The author-reviewer discussion period will close soon. The authors put much efforts for their rebuttal. Please read and respond to authors if you have not done yet.

Thank you.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.