- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes PDCM, which treats batch allocation as the control variable in multi-task LLM reinforcement-learning post-training and combines estimated task-improvement signals with primal-dual updates to improve overall performance while limiting each task’s local counterfactual lag relative to uniform allocation. Its main contribution lies in an online data-allocation mechanism and its application to practical post-training.
Critical strengths
1. The problem is practically relevant, and the proposed intervention is compatible with existing post-training pipelines.
The paper studies an important problem in multi-task post-training: how to allocate limited training budgets across different capabilities. By adjusting batch composition rather than directly modifying task-specific gradients, the proposed approach can be naturally integrated into existing GSPO pipelines. The formulation jointly considers overall improvement and task-level protection, providing a practical and principled solution.
2. The experiments demonstrate consistent improvements across task combinations with multiple random seeds.
The main results show that PDCM consistently outperforms baseline allocation strategies across different task combinations and evaluation settings. For example, on Nemo-Gym, PDCM improves macro-average pass@1 from 59.57% to 63.30% over Uniform allocation, demonstrating that adaptive task allocation provides practical benefits beyond a purely formal optimization framework.
Critical weaknesses
I am less familiar with the theoretical aspects of online convex optimization and the regret analysis, so my comments mainly focus on the empirical validation and experimental design.
W1:
Although the paper evaluates PDCM on two task suites, all main experiments are conducted with a single model (Qwen3-4B) and a single RL post-training algorithm (GSPO). It remains unclear whether the observed improvements are specific to this training configuration or generalize to different model scales, model families, or RL optimization frameworks. The proposed method is presented as a general allocation mechanism for multi-task LLM post-training, but the current evaluation does not sufficiently test this generality.
W2:
Since PDCM is fundamentally an adaptive task allocation method, stronger comparisons against existing online sampling or curriculum strategies are needed to establish that the improvement comes from the proposed formulation rather than generic adaptive scheduling.
Questions for authors
Q1:
The experiments are mainly conducted with Qwen3-4B and GSPO. Could the authors comment on the expected robustness of PDCM across different model scales or RL post-training algorithms? Is the allocation signal used by PDCM expected to remain stable when the underlying optimizer changes?
Q2:
Since PDCM is an adaptive allocation method, could the authors clarify how it compares conceptually and empirically with other adaptive sampling or curriculum-based approaches?
4 comments
Response to W2
We thank the reviewer for this suggestion. To address this concern, we have conducted additional comparisons against two adaptive curriculum learning methods:
- Self-Evolving Curriculum (SEC) (Chen et al., 2025), which formulates curriculum selection as a non-stationary bandit problem and uses absolute advantage as a proxy for learning progress.
- Actor-Curator (AC) (Gu et al., 2026), which uses a policy-improvement bandit to adaptively select training problems.
Both methods adjust training data selection online, providing relevant baselines for evaluating whether PDCM's improvements can be explained by adaptive scheduling alone.
New results on the four-task Nemo-Gym mixture using Qwen3-4B
Pass@1 (%):
| Method | code_gen | IF | workplace | xlam | Total |
|---|---|---|---|---|---|
| PDCM | 32.61 | 53.33 | 82.89 | 84.35 | 253.18 |
| SEC | 30.99 | 52.33 | 82.86 | 81.42 | 247.60 |
| AC | 31.53 | 50.83 | 69.67 | 82.42 | 234.45 |
| Aioli | 35.40 | 47.79 | 82.11 | 82.32 | 247.62 |
| PCGrad | 34.39 | 51.00 | 74.72 | 82.92 | 243.03 |
| Uniform | 34.08 | 47.58 | 81.74 | 74.87 | 238.27 |
| Single | 31.29 | 36.63 | 82.61 | 83.57 | 234.10 |
Pass@4 (%):
| Method | code_gen | IF | workplace | xlam | Total |
|---|---|---|---|---|---|
| PDCM | 40.55 | 61.21 | 88.15 | 84.66 | 274.57 |
| SEC | 39.07 | 61.25 | 87.29 | 82.80 | 270.41 |
| AC | 40.98 | 57.02 | 81.47 | 82.04 | 261.51 |
| Aioli | 43.29 | 52.09 | 87.97 | 84.10 | 267.45 |
| PCGrad | 42.43 | 58.81 | 75.15 | 83.19 | 259.58 |
| Uniform | 41.41 | 54.57 | 84.00 | 80.70 | 260.68 |
| Single | 37.73 | 45.76 | 86.46 | 83.91 | 253.86 |
PDCM achieves the highest aggregate performance across both metrics, outperforming SEC by 5.58 points in pass@1 and 4.16 points in pass@4, and AC by 18.73 and 13.06 points, respectively. Notably, SEC is a competitive baseline, ranking second overall at pass@4, yet PDCM maintains a consistent aggregate advantage.
Why does PDCM differ from generic adaptive scheduling?
While SEC uses absolute advantage to guide curriculum selection and AC optimizes expected policy improvement through a bandit formulation, PDCM explicitly optimizes cumulative task-wise policy improvement subject to a no-task-left-behind constraint. This introduces a dual feedback mechanism that accounts for cumulative task-wise allocation deficits relative to uniform allocation, rather than relying solely on immediate learning signals.
The new comparisons show that adaptive scheduling alone does not fully explain PDCM's performance advantage. Furthermore, our dual-removal ablation in Table 2 provides complementary evidence: removing the dual term reduces total pass@1 from 253.18 to 244.57 and pass@4 from 274.57 to 267.30.
Together, these results provide stronger empirical support for our constrained allocation formulation. PDCM not only improves upon uniform allocation and existing mixing methods, but also outperforms two adaptive curriculum learning approaches designed specifically for RL post-training.
Response to Questions
Response to Q1: Robustness Across Model Scales and RL Algorithms
Please see our response to W1. We have conducted additional Qwen3-8B experiments (Appendix E, Table 6 and Figure 7), demonstrating that PDCM's improvements persist when model size is doubled and the task mixture expands from four to eight heterogeneous domains.
Regarding different RL algorithms, PDCM operates by adjusting batch composition without modifying the underlying policy optimization procedure. Its allocation signal is based on estimated task-wise policy improvement rather than optimizer-specific gradient information. Therefore, the formulation is applicable in principle to other RL algorithms, provided that a suitable policy-improvement estimator is available.
However, we do not assume that the estimated allocation signal remains unchanged when the optimizer changes. Different optimizers can induce different allocation-response functions, which PDCM would need to estimate online. We acknowledge that direct experiments with other optimizers and model families are needed to establish empirical robustness beyond GSPO.
Response to Q2: Comparison with Adaptive Sampling and Curriculum Methods
Please see our response to W2. We have added experiments comparing PDCM against Self-Evolving Curriculum (SEC) and Actor-Curator (AC) on the four-task Nemo-Gym mixture.
PDCM outperforms both methods in aggregate performance, achieving 253.18 total pass@1 and 274.57 total pass@4, compared with 247.60/270.41 for SEC and 234.45/261.51 for AC.
Conceptually, PDCM differs from these approaches by explicitly optimizing cumulative task-wise policy improvement subject to a no-task-left-behind constraint. Its dual variables provide feedback on estimated cumulative constraint violations, rather than using immediate learning signals alone to determine task allocation.
Together with the dual-removal ablation in Table 2, these comparisons provide evidence that PDCM's gains are not solely attributable to generic adaptive scheduling, but also benefit from its constrained allocation mechanism.
Thank you for the detailed responses and additional experiments. The new Qwen3-8B results and the experiments with more heterogeneous task mixtures alleviate my concern about the limited evaluation scope. The comparisons with SEC and AC, together with the dual-removal ablation, also provide stronger empirical support for the proposed constrained allocation mechanism and substantially address my concern regarding comparisons with existing adaptive methods. Generalization across model families and RL algorithms remains unverified, but the authors have clearly acknowledged this limitation. Overall, the additional evidence has improved my assessment of the paper, and I will raise my rating from weak reject to weak accept. Please ensure that the additional results and remaining limitations are clearly presented in the revised manuscript, with the generality claims appropriately scoped to the available evidence.
Response to W1
Response to W1: Generality Across Model Scales and RL Algorithms
We thank the reviewer for raising this concern. We agree that the original experiments were limited to Qwen3-4B with GSPO, and have conducted additional experiments to evaluate whether PDCM's improvements persist at larger model scales.
1. New experiments demonstrate that PDCM's improvements extend to Qwen3-8B.
In Appendix E (Table 6), we evaluate PDCM on Qwen3-8B using the four-task Nemo-Gym mixture, maintaining the same training configuration and budget. PDCM outperforms uniform allocation on all four tasks under both pass@1 and pass@4.
On Qwen3-8B, PDCM improves total pass@1 by 12.14 points and pass@4 by 10.64 points, demonstrating that the benefits of adaptive allocation persist when model size is doubled.
Furthermore, we extend the Qwen3-8B experiments from four to eight heterogeneous task domains (Appendix E, Figure 7). PDCM consistently outperforms uniform allocation across all five mixture sizes, with total pass@1 improvements ranging from 7.4 to 17.3 points. These results provide additional evidence that PDCM remains effective as both model size and task heterogeneity increase.
Furthermore, we extended the Qwen3-8B experiments from four to eight heterogeneous task domains by introducing hermes, api_agent, toolace, and glaive_fc. As shown in Appendix E, Figure 7, PDCM consistently outperforms uniform allocation across all five mixture sizes, with total pass@1 gain margins ranging from 7.4 to approximately 17 points.
Table: Qwen3-8B scaling across heterogeneous task domains (total pass@1).
The table reports cumulative task mixtures, with each row adding one domain to the preceding mixture. The corresponding scaling plot is provided in Appendix E, Figure 7, where the gains are measured relative to the initial model.
Unlike the 18-component Reasoning-Gym experiment, which divides six task families into difficulty tiers, these new experiments scale the Nemo-Gym mixture to eight distinct task domains using the full-Jacobian formulation. This provides additional evidence of scalability in heterogeneous multi-task post-training.
2. PDCM is designed as an allocation mechanism independent of the underlying RL optimizer.
As discussed in Section 3, PDCM operates exclusively by adjusting task proportions in each training batch, without modifying the underlying RL optimization algorithm. Its primal-dual allocation updates depend on estimated task-wise policy improvements and their allocation sensitivities, rather than optimizer-specific gradient operations.
This separation allows the allocation framework to be applied, in principle, to other RL algorithms that provide suitable estimates of policy improvement. Equation (11) uses importance-weighted advantages to estimate these improvements, which is particularly convenient for group-based RL methods such as GRPO and GSPO.
We acknowledge that our current experiments do not directly establish generalization across model families or RL optimization frameworks. Such extensions may require adapting the policy-improvement estimator to the underlying RL algorithm. Nevertheless, the new Qwen3-8B experiments address the model-scale limitation, while the separation between batch allocation and policy optimization provides a clear basis for extending PDCM beyond GSPO.
In summary, our additional experiments establish that PDCM's improvements are not restricted to Qwen3-4B and remain consistent across larger models and more heterogeneous task mixtures. While cross-family and cross-optimizer validation remains outside our current evaluation, the allocation-only design does not require changes to the underlying RL update rule.