acceptodds
Under review as a conference paper at ICLR 2027

Allocate Don't Operate: Online Data Allocation for Multi-Task LLM Post-Training

Abstract

Post-training a large language model rarely targets a single capability: a typical run mixes coding, tool-calling, mathematical reasoning, and instruction following. Classical multi-task learning studies this setting through the lenses of gradient interference and fairness, motivating methods such as gradient surgery and MGDA. These fit post-training poorly. Gradient-surgery methods must materialize per-task gradients at a cost that grows with both the number of tasks and the size of the model, while multi-objective methods target asymptotic balance in runs that stop well before convergence. We instead take the per-task quota of each mini-batch as the only control variable, casting multi-task post-training as online convex optimization: maximize cumulative policy improvement subject to a “no-task-left-behind” constraint. Assuming only directional concavity of the task-improvement surface, that constraint relaxes into a linear one computable from quantities already available at the current allocation. This yields Primal-Dual Curvature Matching (PDCM), a mirror-descent-like allocator driven by first-order estimates of how policy improvement responds to allocation, and unlike allocation methods developed for LLM pre-training it presumes no parametric model of training dynamics. By controlling batch composition alone, PDCM leads the uniform baseline on pass@1 by as much as 6.3% and consistently beats gradient-surgery and parametric mixing methods. Additionally, it scales to mixtures of up to 18 tasks, holding a margin a much as 5.3% over the uniform baseline with an increasing number of tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Post-Training, LLM, RL, Online Optimizaion

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Sp9COpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes PDCM, which treats batch allocation as the control variable in multi-task LLM reinforcement-learning post-training and combines estimated task-improvement signals with primal-dual updates to improve overall performance while limiting each task’s local counterfactual lag relative to uniform allocation. Its main contribution lies in an online data-allocation mechanism and its application to practical post-training.

Critical strengths

1. The problem is practically relevant, and the proposed intervention is compatible with existing post-training pipelines.

The paper studies an important problem in multi-task post-training: how to allocate limited training budgets across different capabilities. By adjusting batch composition rather than directly modifying task-specific gradients, the proposed approach can be naturally integrated into existing GSPO pipelines. The formulation jointly considers overall improvement and task-level protection, providing a practical and principled solution.

2. The experiments demonstrate consistent improvements across task combinations with multiple random seeds.

The main results show that PDCM consistently outperforms baseline allocation strategies across different task combinations and evaluation settings. For example, on Nemo-Gym, PDCM improves macro-average pass@1 from 59.57% to 63.30% over Uniform allocation, demonstrating that adaptive task allocation provides practical benefits beyond a purely formal optimization framework.

Critical weaknesses

I am less familiar with the theoretical aspects of online convex optimization and the regret analysis, so my comments mainly focus on the empirical validation and experimental design.

W1:

Although the paper evaluates PDCM on two task suites, all main experiments are conducted with a single model (Qwen3-4B) and a single RL post-training algorithm (GSPO). It remains unclear whether the observed improvements are specific to this training configuration or generalize to different model scales, model families, or RL optimization frameworks. The proposed method is presented as a general allocation mechanism for multi-task LLM post-training, but the current evaluation does not sufficiently test this generality.

W2:

Since PDCM is fundamentally an adaptive task allocation method, stronger comparisons against existing online sampling or curriculum strategies are needed to establish that the improvement comes from the proposed formulation rather than generic adaptive scheduling.

Questions for authors

Q1:

The experiments are mainly conducted with Qwen3-4B and GSPO. Could the authors comment on the expected robustness of PDCM across different model scales or RL post-training algorithms? Is the allocation signal used by PDCM expected to remain stable when the underlying optimizer changes?

Q2:

Since PDCM is an adaptive allocation method, could the authors clarify how it compares conceptually and empirically with other adaptive sampling or curriculum-based approaches?

4 comments
AuthorsOpenReview ↗

Response to W1

Response to W1: Generality Across Model Scales and RL Algorithms

We thank the reviewer for raising this concern. We agree that the original experiments were limited to Qwen3-4B with GSPO, and have conducted additional experiments to evaluate whether PDCM's improvements persist at larger model scales.

1. New experiments demonstrate that PDCM's improvements extend to Qwen3-8B.

In Appendix E (Table 6), we evaluate PDCM on Qwen3-8B using the four-task Nemo-Gym mixture, maintaining the same training configuration and budget. PDCM outperforms uniform allocation on all four tasks under both pass@1 and pass@4.

ModelPDCM pass@1Uniform pass@1PDCM pass@4Uniform pass@4
Qwen3-4B253.18238.27274.57260.68
Qwen3-8B244.20232.06260.33249.69

On Qwen3-8B, PDCM improves total pass@1 by 12.14 points and pass@4 by 10.64 points, demonstrating that the benefits of adaptive allocation persist when model size is doubled.

Furthermore, we extend the Qwen3-8B experiments from four to eight heterogeneous task domains (Appendix E, Figure 7). PDCM consistently outperforms uniform allocation across all five mixture sizes, with total pass@1 improvements ranging from 7.4 to 17.3 points. These results provide additional evidence that PDCM remains effective as both model size and task heterogeneity increase.

Furthermore, we extended the Qwen3-8B experiments from four to eight heterogeneous task domains by introducing hermes, api_agent, toolace, and glaive_fc. As shown in Appendix E, Figure 7, PDCM consistently outperforms uniform allocation across all five mixture sizes, with total pass@1 gain margins ranging from 7.4 to approximately 17 points.

Table: Qwen3-8B scaling across heterogeneous task domains (total pass@1).

Task mixturePDCMUniformInitial
code_gen, IF, workplace, xlam244.20232.06148.98
+ hermes314.89300.76212.28
+ api_agent396.81379.56294.49
+ toolace463.07455.68352.23
+ glaive_fc557.97549.43449.15

The table reports cumulative task mixtures, with each row adding one domain to the preceding mixture. The corresponding scaling plot is provided in Appendix E, Figure 7, where the gains are measured relative to the initial model.

Unlike the 18-component Reasoning-Gym experiment, which divides six task families into difficulty tiers, these new experiments scale the Nemo-Gym mixture to eight distinct task domains using the full-Jacobian formulation. This provides additional evidence of scalability in heterogeneous multi-task post-training.

2. PDCM is designed as an allocation mechanism independent of the underlying RL optimizer.

As discussed in Section 3, PDCM operates exclusively by adjusting task proportions in each training batch, without modifying the underlying RL optimization algorithm. Its primal-dual allocation updates depend on estimated task-wise policy improvements and their allocation sensitivities, rather than optimizer-specific gradient operations.

This separation allows the allocation framework to be applied, in principle, to other RL algorithms that provide suitable estimates of policy improvement. Equation (11) uses importance-weighted advantages to estimate these improvements, which is particularly convenient for group-based RL methods such as GRPO and GSPO.

We acknowledge that our current experiments do not directly establish generalization across model families or RL optimization frameworks. Such extensions may require adapting the policy-improvement estimator to the underlying RL algorithm. Nevertheless, the new Qwen3-8B experiments address the model-scale limitation, while the separation between batch allocation and policy optimization provides a clear basis for extending PDCM beyond GSPO.

In summary, our additional experiments establish that PDCM's improvements are not restricted to Qwen3-4B and remain consistent across larger models and more heterogeneous task mixtures. While cross-family and cross-optimizer validation remains outside our current evaluation, the allocation-only design does not require changes to the underlying RL update rule.

AuthorsOpenReview ↗

Response to W2

We thank the reviewer for this suggestion. To address this concern, we have conducted additional comparisons against two adaptive curriculum learning methods:

  • Self-Evolving Curriculum (SEC) (Chen et al., 2025), which formulates curriculum selection as a non-stationary bandit problem and uses absolute advantage as a proxy for learning progress.
  • Actor-Curator (AC) (Gu et al., 2026), which uses a policy-improvement bandit to adaptively select training problems.

Both methods adjust training data selection online, providing relevant baselines for evaluating whether PDCM's improvements can be explained by adaptive scheduling alone.

New results on the four-task Nemo-Gym mixture using Qwen3-4B

Pass@1 (%):

Methodcode_genIFworkplacexlamTotal
PDCM32.6153.3382.8984.35253.18
SEC30.9952.3382.8681.42247.60
AC31.5350.8369.6782.42234.45
Aioli35.4047.7982.1182.32247.62
PCGrad34.3951.0074.7282.92243.03
Uniform34.0847.5881.7474.87238.27
Single31.2936.6382.6183.57234.10

Pass@4 (%):

Methodcode_genIFworkplacexlamTotal
PDCM40.5561.2188.1584.66274.57
SEC39.0761.2587.2982.80270.41
AC40.9857.0281.4782.04261.51
Aioli43.2952.0987.9784.10267.45
PCGrad42.4358.8175.1583.19259.58
Uniform41.4154.5784.0080.70260.68
Single37.7345.7686.4683.91253.86

PDCM achieves the highest aggregate performance across both metrics, outperforming SEC by 5.58 points in pass@1 and 4.16 points in pass@4, and AC by 18.73 and 13.06 points, respectively. Notably, SEC is a competitive baseline, ranking second overall at pass@4, yet PDCM maintains a consistent aggregate advantage.

Why does PDCM differ from generic adaptive scheduling?

While SEC uses absolute advantage to guide curriculum selection and AC optimizes expected policy improvement through a bandit formulation, PDCM explicitly optimizes cumulative task-wise policy improvement subject to a no-task-left-behind constraint. This introduces a dual feedback mechanism that accounts for cumulative task-wise allocation deficits relative to uniform allocation, rather than relying solely on immediate learning signals.

The new comparisons show that adaptive scheduling alone does not fully explain PDCM's performance advantage. Furthermore, our dual-removal ablation in Table 2 provides complementary evidence: removing the dual term reduces total pass@1 from 253.18 to 244.57 and pass@4 from 274.57 to 267.30.

Together, these results provide stronger empirical support for our constrained allocation formulation. PDCM not only improves upon uniform allocation and existing mixing methods, but also outperforms two adaptive curriculum learning approaches designed specifically for RL post-training.

AuthorsOpenReview ↗

Response to Questions

Response to Q1: Robustness Across Model Scales and RL Algorithms

Please see our response to W1. We have conducted additional Qwen3-8B experiments (Appendix E, Table 6 and Figure 7), demonstrating that PDCM's improvements persist when model size is doubled and the task mixture expands from four to eight heterogeneous domains.

Regarding different RL algorithms, PDCM operates by adjusting batch composition without modifying the underlying policy optimization procedure. Its allocation signal is based on estimated task-wise policy improvement rather than optimizer-specific gradient information. Therefore, the formulation is applicable in principle to other RL algorithms, provided that a suitable policy-improvement estimator is available.

However, we do not assume that the estimated allocation signal remains unchanged when the optimizer changes. Different optimizers can induce different allocation-response functions, which PDCM would need to estimate online. We acknowledge that direct experiments with other optimizers and model families are needed to establish empirical robustness beyond GSPO.

Response to Q2: Comparison with Adaptive Sampling and Curriculum Methods

Please see our response to W2. We have added experiments comparing PDCM against Self-Evolving Curriculum (SEC) and Actor-Curator (AC) on the four-task Nemo-Gym mixture.

PDCM outperforms both methods in aggregate performance, achieving 253.18 total pass@1 and 274.57 total pass@4, compared with 247.60/270.41 for SEC and 234.45/261.51 for AC.

Conceptually, PDCM differs from these approaches by explicitly optimizing cumulative task-wise policy improvement subject to a no-task-left-behind constraint. Its dual variables provide feedback on estimated cumulative constraint violations, rather than using immediate learning signals alone to determine task allocation.

Together with the dual-removal ablation in Table 2, these comparisons provide evidence that PDCM's gains are not solely attributable to generic adaptive scheduling, but also benefit from its constrained allocation mechanism.

Thank you for the detailed responses and additional experiments. The new Qwen3-8B results and the experiments with more heterogeneous task mixtures alleviate my concern about the limited evaluation scope. The comparisons with SEC and AC, together with the dual-removal ablation, also provide stronger empirical support for the proposed constrained allocation mechanism and substantially address my concern regarding comparisons with existing adaptive methods. Generalization across model families and RL algorithms remains unverified, but the authors have clearly acknowledged this limitation. Overall, the additional evidence has improved my assessment of the paper, and I will raise my rating from weak reject to weak accept. Please ensure that the additional results and remaining limitations are clearly presented in the revised manuscript, with the generality claims appropriately scoped to the available evidence.

Reviewer_PecgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Primal-Dual Curvature Matching (PDCM), an online data-allocation method for multi-task LLM reinforcement-learning post-training. It estimates how task improvements respond to changes in batch composition and adjusts allocation through an exponentiated-gradient update with task-specific dual variables. A concavity assumption converts cumulative lag relative to a uniform-allocation counterfactual into a first-order surrogate constraint. Experiments on Qwen3-4B show aggregate improvements over uniform sampling, PCGrad, and Aioli on agentic and reasoning task mixtures.

Critical strengths

1 Practical motivation and implementation

Controlling batch composition is a useful way to address multi-task training efficiency while retaining the underlying RL optimizer. Reusing existing rollouts and avoiding separate task-gradient storage makes the approach operationally attractive.

  1. Encouraging empirical results.

Experiments use four random seeds, and PDCM achieves the highest aggregate score on both main suites at pass@1 and pass@4. The pass@1 improvements over uniform correspond to approximately 3.73 percentage points in average task accuracy on Nemo-Gym and 1.93 points on ACZ.

  1. Useful component analyses.

The dual-term ablation supports the practical value of the feedback mechanism, while the full-versus-diagonal Jacobian comparison examines the contribution of cross-task information.

Critical weaknesses
  1. The allocation-response estimator is central but insufficiently justified or validated.

Equation (12) estimates each Jacobian entry using pairwise univariate slopes over recent training steps. However, allocation coordinates change together under the simplex constraint, and the policy also evolves across these steps. Consequently, the estimated slopes can conflate cross-coordinate effects and temporal changes with the desired local allocation sensitivity. Theil–Sen robustness to outliers does not resolve this identification problem. The paper does not compare predicted responses against controlled allocation perturbations from fixed checkpoints, nor directly compare its estimated lag with the actual uniform-allocation counterfactual. Such validation is necessary to establish whether the method measures the quantity on which its theoretical interpretation depends.

  1. The concavity evidence and theoretical guarantees do not adequately cover the implemented algorithm.

Appendix B probes ten-step improvement along selected allocation directions at one checkpoint, whereas the assumption concerns one-step improvement over the entire simplex at every round. Constructing an upper concave envelope does not establish that the underlying response is concave; several reported responses are visibly non-monotonic in ways that warrant a more rigorous test.

  1. The evaluation provides limited evidence for generality, scalability, and efficiency.

All experiments use one model, Qwen3-4B, with GSPO in non-thinking mode. The main training configuration uses two H200s, a stated 48-hour budget, and a 600-step configuration, without systematic variation in model size or training budget. Nemo-Gym covers useful heterogeneous capabilities, but three of its four evaluations use in-distribution splits; Reasoning-Gym mainly evaluates procedurally generated tasks. The 18-component experiment comprises six task families divided into difficulty tiers and uses the scalar/diagonal variant, so it does not establish full-Jacobian scalability to 18 heterogeneous tasks. Moreover, the full Jacobian improves Nemo-Gym pass@1 total over the diagonal variant by only 0.36 points, without uncertainty reported for that comparison. Measured overhead and actual compute consumption are also needed to substantiate the efficiency claim.

Questions for authors

1 Can the authors validate both the estimated allocation responses and the uniform-counterfactual lag using controlled one-step experiments from several fixed checkpoints?

2 What conditions make Equation (12) a reliable estimator despite coupled allocation changes and policy drift, and how sensitive are results to the history window and perturbation strength?

8 comments
AuthorsOpenReview ↗

Response to Weakness 1

Response to W1: Validity of the allocation-response estimator and counterfactual-lag approximation

We thank the reviewer for the detailed questions concerning Equation (12). We would like to clarify the theoretical origin of the allocation-response Jacobian, the motivation for our estimation procedure, and the intended interpretation of the uniform-counterfactual lag.

1. The allocation-response Jacobian is derived from our constrained optimization formulation.

Our formulation begins with the task-wise policy improvement Ut,k(p)U_{t,k}(p), defined in Section 2 as the improvement resulting from a joint policy update under allocation pp. Importantly, we do not assume that task improvements are separable across allocation coordinates. Since all tasks share the same policy parameters, training on task jj can affect the improvement of task kk, as explicitly discussed in Section 2.

Starting from the cumulative counterfactual-lag constraint in Equation (2), Assumption 1 provides the first-order upper bound in Equation (3):

Ut,k(u)−Ut,k(pt)≤⟨gt,k,u−pt⟩,U_{t,k}(u)-U_{t,k}(p_t)\leq \langle g_{t,k},u-p_t\rangle,

where gt,k=∇Ut,k(pt)g_{t,k}=\nabla U_{t,k}(p_t). This leads to the constrained optimization problem in Equation (4), whose primal-dual updates yield the allocation score in Equation (8):

st=Gt⊤(1+λt).s_t=G_t^\top(\mathbf{1}+\lambda_t).

Thus, the full allocation-response Jacobian GtG_t arises naturally from the constrained optimization problem, rather than being an assumed model of independent task responses. Its off-diagonal entries explicitly represent cross-task allocation effects.

2. Equation (12) approximates these theoretically defined first-order responses using empirical policy improvement.

Equation (11) estimates the task-wise policy improvement Ut,kU_{t,k} using importance-weighted advantages following the joint policy update. Consequently, each observation U^t,k\hat U_{t,k} reflects policy improvement under the complete task mixture, including cross-task interactions through shared model parameters.

Equation (12) then estimates the entries of GtG_t using coordinate-wise Theil–Sen regression. This choice is motivated by the substantial noise in individual RL updates. Rather than estimating sensitivities from a single observation, we use a historical window of H=10H=10 steps (Appendix C, Table 4). Additionally, Algorithm 1 introduces Dirichlet dithering to provide variation in allocation proportions.

We agree that coordinate-wise regression does not, by itself, eliminate the effects of coupled allocation changes. Likewise, robustness to noisy observations does not automatically eliminate temporal bias. Our estimation procedure relies on the allocation-response function being approximately stable within the short historical window, allowing the observed allocation variation to provide informative local response estimates.

Importantly, Appendix D provides empirical evidence that PDCM remains effective even under a substantial simplification of the Jacobian. Table 5 compares the full-Jacobian estimator against a diagonal variant that discards all off-diagonal entries. The diagonal variant achieves a total pass@1 of 252.82 compared with 253.18 for the full estimator, and 271.51 compared with 274.57 at pass@4. Both substantially outperform the uniform baseline. While this does not directly establish estimator accuracy, it suggests that PDCM's effectiveness does not critically depend on accurately recovering every cross-task interaction.

AuthorsOpenReview ↗

Response to Weakness 1 (Contd.)

3. The uniform-counterfactual lag is intentionally controlled through a tractable first-order surrogate rather than estimated directly.

The reviewer requests direct validation of the uniform-counterfactual lag. However, the difficulty of measuring this quantity is precisely the motivation for our formulation. Equation (2) requires evaluating counterfactual policy improvements under uniform allocation from each intermediate policy state. Directly measuring these quantities would require additional training updates under uniform allocation at every step, incurring substantial computational overhead. We therefore introduce the concavity assumption to derive the conservative first-order upper bound in Equation (3):

Ut,k(u)−Ut,k(pt)≤⟨gt,k,u−pt⟩,U_{t,k}(u)-U_{t,k}(p_t)\leq \langle g_{t,k},u-p_t\rangle,

which leads to the tractable constraint in Equation (4).

Importantly, our goal is not to reconstruct the unobserved counterfactual exactly, but to control its cumulative violation through a tractable surrogate. Under the assumptions of our theoretical analysis in Appendix E, controlling this first-order constraint provides an upper bound on cumulative counterfactual-lag violations. The dual variable λt,k\lambda_{t,k} is therefore not intended as a direct predictor of the evaluation gap against a separately trained uniform baseline. Rather, it accumulates estimated first-order constraint violations and adjusts future allocations through Equation (8).

To investigate this mechanism empirically, we measure the correlation between λt,k\lambda_{t,k}, averaged over 20-step windows, and the observed evaluation gap relative to uniform allocation:

TaskQwen3-4BQwen3-8B
IF-0.69-0.97
xlam-0.22-0.52
workplace-0.56-0.24
code_gen+0.95+0.68

For three of the four tasks, we observe negative correlations across both model sizes. These results are consistent with the intended feedback mechanism, in which increased dual pressure is associated with reduced evaluation lag. The positive correlations for code_gen indicate that this relationship is not universal. We emphasize that these correlations are not intended to establish the accuracy of the counterfactual estimator, but to provide an empirical diagnostic of the dual control mechanism.

More importantly, our existing ablations directly demonstrate the practical value of this control mechanism. In Table 2, removing the dual term reduces total pass@1 from 253.18 to 244.57 and pass@4 from 274.57 to 267.30 on Nemo-Gym. Removing the constraint entirely through OSMD reduces performance further. Furthermore, Table 1 shows that PDCM outperforms uniform allocation while maintaining competitive performance across individual tasks, consistent with the intended no-task-left-behind objective.

Taken together, the theoretical upper bound, the observed relationship between dual pressure and evaluation lag, and the ablation results support the practical effectiveness of controlling the first-order surrogate. We acknowledge that these results do not constitute direct validation of the unobserved counterfactual or establish unbiased Jacobian estimation.

AuthorsOpenReview ↗

Response to Weakness 2

Response to W2: Empirical Validation of Concavity and Scope of Theoretical Guarantees

We thank the reviewer for raising this concern. We acknowledge that Appendix B provides limited evidence for Assumption 1, as it examines selected allocation directions at a single checkpoint. We would like to present additional experimental evidence, clarify the motivation for our measurement procedure, and distinguish our theoretical assumptions from practical implementation.

1. New experiments provide stronger evidence for local concavity under cross-task allocation changes.

To address the limitations of Appendix B, we conducted additional experiments in Appendix E (Figure 8), directly examining allocation responses under joint perturbations of multiple task proportions.

Starting from a common checkpoint at step 300 of Nemo-Gym, we perturb the allocation vector along three directions: workplace–IF, code_gen–xlam, and (workplace, code_gen)–(IF, xlam). Each perturbation transfers an allocation mass of 0.10, and task-wise improvements are measured after 10 GSPO steps.

Unlike Appendix B, these experiments explicitly examine joint allocation changes. We assess concavity using the discrete second difference:

Dk,d2=Uk(p+δd)−2Uk(p)+Uk(p−δd).D^2_{k,d}=U_k(p+\delta d)-2U_k(p)+U_k(p-\delta d).

Of the 12 task-direction combinations, 11 satisfy Dk,d2≤0D^2_{k,d}\leq 0, with only code_gen under code_gen–xlam reallocation showing a positive value (D2=0.0070D^2=0.0070). This provides additional evidence for local concavity without relying on a constructed concave envelope.

We also clarify that non-monotonicity does not imply non-concavity. A concave response may initially increase and subsequently decrease. The relevant condition is whether the response lies above the linear interpolation between neighboring allocations, rather than whether it increases monotonically.

2. The 10-step probe reflects a deliberate bias–variance tradeoff consistent with PDCM's practical estimator.

We acknowledge that our theoretical formulation defines one-step policy improvement, whereas the experiments measure improvement over 10 steps. This choice is intentional: individual policy updates typically induce changes too small relative to evaluation noise to reliably distinguish allocation effects or assess concavity.

Importantly, PDCM itself does not estimate allocation sensitivities from isolated updates. Equation (12) applies Theil–Sen regression over a historical window of H=10H=10 steps (Appendix C, Table 4), relying on approximate local stability to extract informative first-order responses from noisy observations.

Although the perturbation experiment holds allocations fixed for 10 steps while PDCM regresses observations collected across 10 consecutive updates, both address the difficulty of extracting reliable signals from individual updates. Thus, the 10-step probe represents a deliberate bias–variance tradeoff at a timescale relevant to PDCM's practical estimation procedure. We acknowledge that 10-step concavity does not formally imply one-step concavity.

3. The theoretical guarantees provide sufficient conditions, while experiments evaluate the practical approximation.

We agree that Theorem 1 and Appendix F analyze an idealized full-Jacobian PDCM under concavity, boundedness, and dual-stability assumptions, whereas the implementation relies on estimated Jacobians. The theoretical guarantees therefore do not automatically extend to the implemented algorithm without additional estimation-error assumptions.

However, the counterfactual-lag bound in Equation (3),

Ut,k(u)−Ut,k(pt)≤⟨gt,k,u−pt⟩,U_{t,k}(u)-U_{t,k}(p_t)\leq\langle g_{t,k},u-p_t\rangle,

only requires the corresponding first-order upper-bound inequality along the direction from ptp_t to uniform allocation uu. This is weaker than global concavity over the entire simplex, which is used for the regret guarantee against arbitrary fixed feasible allocations.

Our experiments therefore test whether local concavity is a reasonable practical approximation, rather than claiming to verify global concavity at every round. Moreover, Appendix E (Table 6) shows that PDCM outperforms uniform allocation on Qwen3-8B by 12.14 points in total pass@1 and 10.64 points in pass@4, improving all four tasks.

In summary, the new experiments strengthen the empirical support for local concavity, while the 10-step measurement horizon reflects the statistical requirements of practical estimation. We emphasize that Assumption 1 provides sufficient theoretical conditions, whereas our experiments demonstrate the effectiveness of the approximate implementation without claiming that global concavity holds exactly throughout training.

Reviewer_sYtqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PDCM adjusts task proportions during multi-task RL post-training. It estimates how each task responds to changes in the data mixture and uses these estimates, together with a task-lag penalty, to allocate the next batch. The underlying GSPO update is retained. Experiments use Qwen3-4B on agentic and reasoning tasks, with four random seeds. Compared with complete training runs using uniform allocation, the method achieves pass@1 improvements.

Critical strengths
  1. Batch composition is a practical control variable for multi-task training. PDCM can adjust the training emphasis without storing and manipulating separate task gradients.
  2. The paper compares against complete uniform-allocation runs and other methods, and reports four-seed results. Gains on both task suites and the dual-removal ablation provide evidence that the allocation strategy is useful in the tested settings.
Critical weaknesses
  1. Equation (12) estimates each allocation effect separately, although task proportions and model states change together. The resulting slopes may mix allocation effects with training progress. Direct validation of the estimated Jacobian is needed.
  2. The paper lacks a comparison with a well-chosen non-uniform fixed mixture, making the benefit of online adaptation difficult to isolate. Clearer uncertainty estimates and measured overhead would also help assess the practical gains.
Questions for authors
  1. How does PDCM compare with complete training under a reasonable non-uniform fixed mixture, selected using prior knowledge or independent validation data? Will fixing a well-chosen non-uniform fixed mixture be much easier to use?
  2. What is the measured overhead relative to uniform allocation, including the extra forward pass and allocation estimation? Please report the computation overhead and wall-clock time consumed, and clarify how the stated 48-hour budget relates to the 600-step configuration.
6 comments
AuthorsOpenReview ↗

Response to W1: Reliability of the allocation-response estimator

We thank the reviewer for raising this concern. We would like to clarify the theoretical motivation of Equation (12), the rationale for its estimation procedure, and the distinction between cross-task interactions and temporal changes in the response function.

First, the Jacobian is derived from our constrained optimization formulation rather than an assumption about independent allocation effects. Starting from the cumulative counterfactual-lag constraint in Equation (2), Assumption 1 gives the first-order bound in Equation (3). The resulting constrained optimization in Equation (4) leads to the primal-dual score in Equation (8), st=Gt⊤(1+λt)s_t = G_t^\top(\mathbf{1}+\lambda_t), where GtG_t is the full allocation-response Jacobian defined in Equation (6). This derivation does not assume task-wise separability: each task's improvement may depend on the entire allocation vector, and the off-diagonal entries of GtG_t represent cross-task responses.

Second, Equation (12) is a practical estimator of this theoretically derived quantity. Equation (11) estimates task-wise improvement Ut,kU_{t,k} using importance-weighted advantages following the joint policy update. Therefore, each observed response reflects training on the complete task mixture. Equation (12) uses coordinate-wise Theil–Sen regression to estimate allocation-response slopes from these observations. We employ a historical window of H=10H=10, rather than estimating responses from individual updates, because one-step policy improvement is highly noisy. We agree that the accuracy of these estimates depends on the local stability of the allocation-response function. In particular, correlated allocation changes and temporal policy drift can affect coordinate-wise slope estimates, even when the underlying improvement observations are accurate. Our intended approximation is that the allocation-response function remains sufficiently stable over the short history window for the resulting slopes to provide useful local first-order information.

Third, our ablation study provides empirical evidence that PDCM is robust to approximations of the full Jacobian. In Appendix D (Table 5), we compare the full-Jacobian PDCM against a diagonal variant that discards all off-diagonal entries, retaining only the estimated response of each task to its own allocation. Despite this substantial simplification, the diagonal variant achieves a total pass@1 of 252.82 compared with 253.18 for the full estimator, and 271.51 compared with 274.57 at pass@4. Both variants substantially outperform the uniform baseline. These results suggest that PDCM's effectiveness does not critically depend on accurately recovering every cross-task interaction. Rather, the algorithm can obtain strong performance from approximate local allocation-response information, even when cross-task effects are not explicitly modeled. While this ablation does not directly establish the accuracy of Equation (12), it provides empirical support for the practical robustness of our approach to Jacobian approximation.

AuthorsOpenReview ↗

Response to W2: Comparison against fixed mixtures

Response: Comparison with Non-Uniform Fixed Allocations

We thank the reviewer for this suggestion. We agree that a carefully selected non-uniform fixed mixture provides an additional point of comparison. To address this concern, we conducted a new experiment comparing PDCM against a fixed mixture constructed from PDCM's own time-averaged allocations, under the same training budget.

First, we provide a new controlled comparison on ACZ. PDCM's time-averaged allocation on ACZ is (0.377,0.331,0.292)(0.377, 0.331, 0.292) for ARC-1D, Countdown, and Zebra, respectively. We train Qwen3-4B using this exact mixture held fixed throughout training, under the same 48-hour wall-clock budget as Table 1.

New results: ACZ pass@1 (%)

TaskPDCMUniformFixed mixture
ARC-1D66.3361.2856.40
Countdown91.8391.0692.39
Zebra91.0891.1287.80
Total249.24243.46236.59

The fixed mixture underperforms PDCM by 12.65 points and even underperforms uniform allocation by 6.87 points. Importantly, the fixed mixture uses PDCM's own observed time-averaged task proportions. Therefore, the difference cannot be explained solely by the average allocation proportions: the temporal distribution of the training budget also matters.

Second, PDCM's time-averaged allocations on Nemo-Gym are nearly uniform. On Qwen3-4B, the observed proportions are (0.251,0.242,0.253,0.254)(0.251, 0.242, 0.253, 0.254), compared with the uniform mixture (0.25,0.25,0.25,0.25)(0.25, 0.25, 0.25, 0.25). Thus, PDCM achieves higher performance than uniform allocation despite assigning nearly identical average training proportions to each task. This suggests that its advantage arises primarily from when the training budget is allocated, rather than simply how much each task receives overall.

This supports our central motivation that task-wise learning opportunities change throughout training, making a static allocation insufficient to capture their temporal dynamics. PDCM responds to these changes by reallocating its budget online. Figure 5 illustrates this behavior on Nemo-Gym, where the most heavily funded task changes across training stages in response to evolving task-wise improvement signals.

Together, the new fixed-mixture experiment on ACZ and the near-uniform time-averaged allocations on Nemo-Gym provide complementary evidence that PDCM's advantage comes from adaptive scheduling rather than merely selecting a favorable static mixture. While this does not rule out the existence of another well-tuned fixed allocation, it demonstrates that even the fixed mixture derived from PDCM's own allocation history cannot reproduce its performance.

AuthorsOpenReview ↗

Response to Questions

Q1: How does PDCM compare with complete training under a reasonable non-uniform fixed mixture, selected using prior knowledge or independent validation data? Will fixing a well-chosen non-uniform fixed mixture be much easier to use? See above.

Q2: What is the measured overhead relative to uniform allocation, including the extra forward pass and allocation estimation? Please report the computation overhead and wall-clock time consumed, and clarify how the stated 48-hour budget relates to the 600-step configuration.

We thank the reviewer for requesting clarification on computational overhead and the training budget.

First, PDCM introduces minimal computational overhead compared with uniform allocation. The additional forward pass required for policy improvement estimation accounts for less than 2% of total training time, while the cost of allocation estimation is negligible in comparison. Moreover, this overhead is often offset by PDCM dynamically shifting the batch composition toward tasks with shorter and easier rollout times at the start of training. Consequently, the additional computation does not necessarily translate into increased wall-clock time per training step.

Second, we use a fixed wall-clock budget rather than a fixed number of training steps to ensure a fair comparison. Since different task mixtures have different rollout costs, fixing the number of steps would not ensure equal computational budgets. Instead, all main experiments are conducted under the same 48-hour wall-clock budget on two H200 GPUs. In practice, experiments complete approximately 420–440 training steps within this budget, with differences in step counts remaining within 5% across methods. This indicates that PDCM's additional forward pass does not lead to a substantial reduction in training throughput.

Finally, the 600 steps reported in Appendix C represent the configured maximum training horizon, not the number of steps actually executed. This configuration is used for the training schedule, such as learning rate warmup, whereas the actual stopping criterion is the 48-hour wall-clock budget. We will clarify this distinction in the revised manuscript to avoid confusion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.