acceptodds
Under review as a conference paper at ICLR 2027

ProCore: Protecting Task-Specific Gradient Cores for Multi-Task RLVR

Abstract

Multi-task RLVR enables a single model to jointly learn multiple tasks. We find that each task’s gradient is highly concentrated, with a small subset of components accounting for most of the gradient norm. We term them gradient cores. Existing methods dynamically adjust task sampling ratios or weights using reward signals or advantage estimates to balance learning progress across tasks. However, they lack explicit protection for gradient cores during gradient aggregation and struggle to characterize each task’s remaining learning potential. Accordingly, we propose ProCore, a plug-and-play method for multi-task RLVR. ProCore identifies gradient cores by magnitude within each Transformer block and partitions them into exclusive and shared cores based on dimension overlap across tasks. It computes a Remaining Potential Score (RPS) from the gap between current pass@1 and an estimated performance ceiling derived from pass@k. Building on the original mixed gradient, ProCore directly enhances exclusive cores and dynamically weights shared cores using RPS, giving greater protection to tasks with more remaining potential. We evaluate ProCore on eight heterogeneous reasoning tasks and nine benchmarks spanning mathematics, coding, and scientific QA. ProCore achieves higher average scores than all baselines across both evaluation settings and further improves DAPO and MT-GRPO when combined with them.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multi-Task RLVR, Protection of Gradient Cores, Remaining Learning Potential Estimation

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_bsWfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates multi-task RLVR at the element level. Specifically, it first argues that RLVR task gradients are rather highly concentrated. This motivates this paper to first identify the task gradient cores and then partition them into shared and exclusive ones. Thereafter, it employs an RPS score, which is estimated from the ratio between Pass@1 and Pass@5 of past rollout scores, to reweight task gradients. Extensive experiments conducted on multiple datasets across models have demonstrated the effectiveness of the proposed method.

Critical strengths
  1. The motivation of this paper is rather clear and easy to follow.

  2. The proposed solution is technically sound and quite simple.

  3. This paper is well-written, with a clear structure.

Critical weaknesses
  1. All designs are quite straightforward and intuitive. From my side, they might lack novelty on the technology level. And I believe the proposed solution is not strong corrective with the high concentration observation; they appearly could be applied to more general scenarios. I also attribute the performance gains to the element-level operations rather than to what the authors claimed.

  2. Since ProCore operates at the element-level, a complexity analysis and comparison is required.

  3. In Figure 3, how are exclusive cores enhanced? According to my understanding, only shared cores are weighted; why are exclusive cores not specially weighted?

  4. The choice of βi(t)\beta_i(t) is too intuitive, and it could be easily uneffective when both Pass@1 and Pass@5 are pool during the early training stage.

Questions for authors

Please refer to the Critical Weaknesses. Besides, no limitations are explicitly claimed in this paper.

5 comments
AuthorsOpenReview ↗

Response to Reviewer bsWf—Part1/4

All designs are quite straightforward and intuitive. From my side, they might lack novelty on the technology level. And I believe the proposed solution is not strong corrective with the high concentration observation; they appearly could be applied to more general scenarios. I also attribute the performance gains to the element-level operations rather than to what the authors claimed.

Thank you very much for your valuable comment. We would like to further clarify the contributions of ProCore and its connection to the observed gradient concentration.

  1. Technical novelty. ProCore introduces a mechanism for identifying and protecting task-specific gradient cores. Beyond the original mixed gradient, ProCore retains the high-magnitude gradient components of each task, distinguishes exclusive and shared cores based on cross-task overlap at each dimension, and uses RPS to determine the protection weights for shared cores. Consequently, even within the same task, different dimensions receive different treatments depending on their core membership. This fine-grained, dimension-level protection mechanism cannot be achieved solely through task-level sampling ratio or weight adjustments (e.g., FAMO, SEC, DADS, DynaSample, and MT-GRPO). Furthermore, unlike multi-task optimization methods such as PCGrad and CAGrad, which require storing and processing the full gradient of each task, ProCore retains only the indices and values of approximately 1.1% of the gradient components for each task after gradient computation, for subsequent protection and enhancement, without needing to store all task gradients simultaneously. This substantially reduces memory overhead and the cost of subsequent gradient processing. Taking PCGrad as an example, to avoid OOM issues, we offload each task's full gradient to CPU memory during training and transfer it back to the GPU as needed for gradient projection and aggregation. This incurs substantial memory and data-transfer overhead, making PCGrad approximately 1.67 times slower to train than ProCore.

  2. Gradient concentration directly determines the protection targets and scope of ProCore. We find that the top 1% of gradient components by magnitude account for an average of 87.3% of each task's gradient norm. This indicates that the primary gradient signals of each task are concentrated in a small subset of dimensions. Based on this observation, ProCore does not uniformly modify the entire gradient. Instead, it restricts additional gradient protection to a small subset of core dimensions. Specifically, within each Transformer block, we select approximately 1.1% of the gradient components with the largest magnitudes as gradient cores. Building on the original mixed gradient, we directly enhance exclusive cores and apply RPS-weighted enhancement to shared cores, while leaving non-core dimensions unchanged. Therefore, the gradient concentration observation directly guides ProCore's core selection and selective enhancement strategy, enabling it to specifically protect the primary gradient signals of each task.

  3. The performance gains are not solely attributable to element-wise operations. In Table 3 of the paper, both ablation variants retain the element-wise enhancement operation but modify either the core selection strategy or the computation of shared-core gradient weights. Replacing magnitude-based core selection with random selection at the same ratio reduces the average scores by 1.6 and 3.3 percentage points on Qwen3-4B and Qwen3-8B, respectively. Replacing dynamic RPS-based weights with equal weights leads to further reductions of 0.9 and 1.6 percentage points, respectively. Moreover, we conducted additional experiments using Pass@1 and the Pass@K–Pass@1 gap to compute the weights for shared-core gradients, while retaining the element-wise operations. The results are presented in Table 1. ProCore using RPS achieves the highest average performance. Overall, these results demonstrate that ProCore's performance gains cannot be attributed solely to element-wise operations.

    Table 1. Effect of different shared-core weighting strategies on ProCore performance.

    MethodsARC1DCountdownCodeIORubikSokobanZebraWordLadderBitwiseAVG
    ProCore (Pass@1)41.557.032.020.514.038.541.565.038.8
    ProCore (Pass@K-Pass@1)38.059.532.531.59.036.534.573.039.3
    ProCore (RPS)34.057.033.538.529.540.536.567.042.1
AuthorsOpenReview ↗

Response to Reviewer bsWf—Part2/4

Since ProCore operates at the element-level, a complexity analysis and comparison is required.

Thank you very much for your valuable suggestion. Our response is provided below.

  1. Theoretical Complexity. Let N denote the number of tasks and D denote the number of model parameters. We focus on the time and space complexity of gradient processing and aggregation. ProCore identifies gradient cores for each task and applies weighted enhancement to the selected approximately 1.1% of gradient components, with a time complexity of O(ND). In our implementation, although ProCore sequentially accumulates the base gradients from individual tasks, it retains only the indices and values of each task's gradient cores, without simultaneously storing the full gradients of all tasks. Its space complexity is therefore O(D+0.011ND). In addition, RPS reuses reward statistics already collected during training and requires no additional sampling. In comparison, PCGrad performs pairwise gradient projections across the full model parameter space, with a time complexity of O(N²D) for gradient processing. CAGrad constructs an inter-task gradient inner-product matrix and further optimizes the aggregation weights. The complexity of all four methods is summarized in Table 2.

    Table 2. Time and space complexity of gradient processing and aggregation

    MethodTime ComplexitySpace Complexity
    Mixed TrainingO(D)O(D)O(D)O(D)
    PCGradO(N2D)O(N^2D)O(ND)O(ND)
    CAGradO(N2D)O(N^2D) + Weight Optimization OverheadO(ND+N2)O(ND+N^2)
    ProCoreO(ND)O(ND)O(D+0.011ND)O(D+0.011ND)
  2. Actual Runtime Overhead. We conducted experiments using Qwen3-4B on two NVIDIA A100 80GB GPUs, running Mixed Training, PCGrad, CAGrad, and ProCore under identical training configurations. We measured the average total runtime per step over the first three complete training steps. As shown in Table 3, ProCore increases the per-step training time by only 2.53% compared with Mixed Training. Its runtime is also substantially lower than that of PCGrad and CAGrad, demonstrating that ProCore can process and aggregate task gradients with relatively little additional runtime overhead.

    Table 3. Runtime comparison of different methods on Qwen3-4B.

    MethodTraining Step Time (s/step)
    Mixed Training555.90
    PCGrad951.97
    CAGrad787.33
    ProCore569.95
AuthorsOpenReview ↗

Response to Reviewer bsWf—Part3/4

In Figure 3, how are exclusive cores enhanced? According to my understanding, only shared cores are weighted; why are exclusive cores not specially weighted?

Thank you for your comment. First, we would like to clarify that "Direct Enhancement" in Figure 3 refers to directly enhancing exclusive cores, rather than leaving them unweighted. Specifically, for a gradient component selected as a core only by task ii, ProCore directly adds ρ\rho times the corresponding task-specific core gradient to the original mixed gradient, i.e., gfinal(p)=gbase(p)+ρgi(p)g_{\mathrm{final}}(p)=g_{\mathrm{base}}(p)+\rho g_i(p). Here, ρ\rho is the core enhancement coefficient, which is set to 0.1 in our main experiments. We do not apply additional RPS weighting to exclusive cores because an exclusive core is selected by only one task, so there is no need to allocate protection weights among multiple tasks. According to Eq. (10), when Sp={i}S_p=\{i\}, the normalized weight is naturally ωi(p)=1\omega_i(p)=1. In contrast, for shared cores, multiple tasks select the same component as a core, so their relative protection weights need to be dynamically determined based on their RPS values. Therefore, "Direct Enhancement" and "RPS-based Weight" in Figure 3 correspond to direct enhancement of exclusive cores and RPS-weighted enhancement of shared cores, respectively. Both operations enhance the core gradients on top of the original mixed gradient.

The choice of βi(t)\beta_i(t) is too intuitive, and it could be easily uneffective when both Pass@1 and Pass@5 are pool during the early training stage.

Thank you for your comment. We provide three responses to address this concern.

  1. First, we would like to clarify that our method uses Pass@8 rather than Pass@5. Specifically, at each training step, we generate 8 responses for each question and compute the mean Pass@1 and Pass@8 across all questions in that step to obtain the Pass@1 and Pass@8 for the current step, respectively. We set βi(t)\beta_i(t) to the maximum Pass@8 over the most recent 5 training steps as the empirical performance ceiling, where 5 denotes the history window size. Furthermore, An unusually high score on an individual sample has limited influence on the overall estimate, since Pass@1 and Pass@8 are computed over all training data in each step.

  2. Second, when calculating RPS, the numerator is the difference between the empirical performance ceiling and the average Pass@1 over the three most recent training steps, while the denominator is the difference between the same ceiling and the initial Pass@1 before training. Therefore, even if both Pass@1 and Pass@8 are low during early training, RPS will remain valid as long as the model's performance during training exceeds its performance before training. We also clip RPS to ensure that its value remains within the range of [0, 1]. Figure 4 in our paper shows the evolution of RPS during training, demonstrating that no apparent numerical instability occurs.

  3. Finally, previous studies have used Pass@K as a reference for the upper bound of model capability [1][2][3][4]. In designing ProCore, we use the maximum Pass@K over recent training steps as the reference upper bound, with the aim of preserving the best performance the model has recently achieved and using it to estimate how much room for improvement remains on the current task. If we instead use a moving average or other smoothing strategies, the estimated upper bound may decrease when task performance drops, potentially leading us to prematurely conclude that a task has reached its upper bound and consequently reducing its weight. In addition, we conducted experiments on Qwen3-4B using the average Pass@8 over recent training steps as the upper bound for the current step. The results are presented in Table 4. ProCore using the maximum Pass@8 over recent training steps as the upper bound achieves scores no lower than those of the average-based variant on six tasks, while also achieving better overall performance, further validating the effectiveness of our original design.

    Table 4: Comparison of different strategies for estimating the reference upper bound.

    MethodARC1DCountdownCodeIORubikSokobanZebraWordLadderBitwiseAVG
    ProCore (Avg)41.054.033.520.021.540.536.571.039.8
    ProCore (Max)34.057.033.538.529.540.536.567.042.1

References:

[1] Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (Yue et al., NeurIPS 2025.)

[2] Reasoning with Exploration: An Entropy Perspective (Cheng et al., AAAI 2026.)

[3] Beyond Pass@1: Self-Play with Variational Problem Synthesis Sustains RLVR (Liang et al., ICLR 2026.)

[4] Weight Ensembling Improves Reasoning in Language Models (Dang et al., COLM 2025.)

Reviewer_uRhbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper the authors propose ProCore, a gradient aggregation method for multi-task RLVR. The main observation is that each task-specific gradient is highly concentrated, with a small fraction of scalar gradient entries accounting for most of its norm. ProCore identifies these high-magnitude entries within each Transformer block and selectively enhances them on top of the standard mixed training gradient. Entries selected by only one task are enhanced directly, while entries selected by multiple tasks are weighted using a remaining potential score derived from the gap between recent pass@1 and pass@k.

The method is evaluated on 8 reasoning tasks on Qwen3-4B and Qwen3-8B, and on 9 broader benchmarks covering mathematics, coding, and scientific QA. ProCore improves the average performance over mixed training and achieves the best reported average among the evaluated baselines. The paper also provides component ablations, hyperparameter studies, runtime measurements, and experiments combining ProCore with DAPO and MT-GRPO.

Critical strengths
  1. The proposed gradient aggregation mechanism is simple and to the best of my knowledge, reasonably novel. Instead of changing the weight of an entire task gradient, ProCore operates on a small set of high-magnitude gradient entries within each transformer block. This provides a more fine-grained way to preserve task-specific updates and is clearly different from standard task weighting or full-gradient surgery.

  2. The main empirical gains are meaningful and reasonably well supported. On the main 8 task setting, ProCore improves over mixed training by 4.2 points on Qwen3-4B and 3.4 points on Qwen3-8B, with results averaged over three seeds. It also achieves the highest average among a fairly broad set of baselines. The additional evaluations on standard mathematics, coding, and scientific QA benchmarks, as well as the improvements when combined with DAPO and MT-GRPO, make the results more convincing than a narrow proof of concept.

Critical weaknesses
  1. The interpretation of RPS as remaining learning potential is not directly validated. The gap between pass@1 and pass@k is a plausible measure of correct but unreliable responses. However, the paper does not show that RPS at a given point predicts how much a task can improve with further training. Figure 4 only visualizes RPS during ProCore training, where RPS itself influences the following updates. Therefore, the usefulness of RPS-based weighting is supported, but its interpretation as remaining learning potential is less clear.

  2. Some important design choices and practical aspects receive limited empirical analysis. The RPS formulation is compared with equal weighting, but not with simpler alternatives. The evaluation covers several Qwen3 configurations, but no second model family. Peak memory is also not reported, and the closely related MGS method is discussed but not experimentally compared. These points limit the strength of the broader generality and practicality claims, although the main performance results themselves are reasonably extensive.

Questions for authors
  1. RPS validation and design: Does RPS measured at a given checkpoint predict the subsequent improvement of a task under a fixed additional training budget? Also, could the authors compare the proposed normalized score against other alternatives, such as the unnormalized pass@k and pass@1 gap or the recent change in pass@1? This would help clarify both the interpretation of RPS and the importance of its specific formulation.

  2. Generality and relation to MGS: The experiments cover multiple Qwen3 sizes, but all models belong to the same family. Do the authors expect the observed gradient concentration and the selected hyperparameters to transfer to other model families? In addition, Modular Gradient Surgery also uses task-specific gradients within transformer modules to address multi-domain RL interference. Could the authors clarify the main differences from MGS and explain whether a direct comparison is feasible?

  3. Memory and implementation: What is the peak GPU memory overhead of ProCore compared with mixed training? Please also clarify how the computational and memory costs are expected to scale with the number of tasks.

  4. Shared-core behavior: Shared cores are defined by overlap of high-magnitude parameter indices, while the corresponding task gradients may be aligned or conflicting. Could the authors report the prevalence of these two cases and clarify how ProCore affects each?

6 comments
AuthorsOpenReview ↗

Response to Reviewer uRhb—Part1/5

The interpretation of RPS as remaining learning potential is not directly validated. The gap between pass@1 and pass@k is a plausible measure of correct but unreliable responses. However, the paper does not show that RPS at a given point predicts how much a task can improve with further training. Figure 4 only visualizes RPS during ProCore training, where RPS itself influences the following updates. Therefore, the usefulness of RPS-based weighting is supported, but its interpretation as remaining learning potential is less clear.

Thank you very much for your valuable comment. We provide three responses to address this concern.

  1. First, we sincerely apologize for the misunderstanding caused by our explanation of RPS in the paper. We designed the RPS primarily to measure the learning progress of each task and prioritize the protection of tasks that learn more slowly during multi-task training. It is not intended to predict the exact amount of improvement for each task. We generally believe that tasks with slower learning progress have greater remaining learning potential, which is why we named RPS after remaining potential score. We will provide a more detailed explanation of RPS in the paper. Furthermore, to help you better understand RPS, we elaborate on its design motivation below.

  2. Motivation behind RPS. During multi-task RLVR, different tasks learn at different speeds. We aim to prioritize protecting tasks that make slower progress. However, the learning progress of individual tasks is difficult to characterize. Directly using rewards to measure learning progress may overlook tasks that make slow progress but still have room for improvement. Previous studies have found that in single-task RLVR, Pass@1 and Pass@K gradually converge as training proceeds. We observe the same phenomenon for individual tasks in multi-task RLVR. Therefore, we use the Pass@K at each step as the reference upper bound and measure a task's learning progress based on the gap between Pass@1 and this upper bound. Since different tasks have different initial performance levels and reference upper bounds, directly using the gap between Pass@1 and Pass@K makes it difficult to measure the relative learning progress across tasks and may overemphasize certain tasks, thereby affecting the balance of multi-task learning. Therefore, we use the difference between the current reference upper bound and the initial Pass@1 (i.e., the performance before training) as the denominator, and the difference between the current reference upper bound and the current Pass@1 as the numerator. This gives us the normalized learning progress metric, RPS. At initialization, the RPS of all tasks is set to 1. As shown in Figure 4 of our paper, the RPS of each task gradually approaches 0 as training proceeds, but the rates of decrease differ substantially across tasks, further reflecting the differences in their learning speeds.

  3. Furthermore, although RPS is not intended to predict the exact amount of future improvement, it can still provide an estimate of the relative improvement potential across tasks. Tasks with higher RPS values tend to achieve larger relative improvements in success rates with further training. Taking the 45th training step during ProCore (Qwen3-4B) training as an example, the tasks ranked in descending order of RPS at this step are Sokoban, Rubiks, Countdown, ARC1D, WordLadder, CodeIO, Bitwise, and Zebra, with corresponding RPS values of 0.840, 0.745, 0.716, 0.708, 0.693, 0.585, 0.574, and 0.398, respectively. This indicates that Sokoban has the slowest learning progress and therefore should receive greater protection in subsequent training. After another 15 training steps, the relative improvements in success rates across all tasks are reported in Table 1. We observe that the ranking of relative improvements in success rates is consistent with the RPS ranking, with Sokoban achieving the largest relative improvement and Zebra the smallest.

    Table 1: Consistency between the RPS ranking and the ranking of relative improvements in task success rates.

    SokobanRubiksCountdownARC1DWordLadderCodeIOBitwiseZebra
    45 steps4.59.037.024.528.528.554.038.5
    60 steps8.012.042.528.032.030.556.539.5
    Relative Improvement+77.8%+33.3%+14.9%+14.3%+12.3%+7.0%+4.6%+2.6%
AuthorsOpenReview ↗

Response to Reviewer uRhb—Part2/5

Also, could the authors compare the proposed normalized score against other alternatives, such as the unnormalized pass@k and pass@1 gap or the recent change in pass@1? This would help clarify both the interpretation of RPS and the importance of its specific formulation.

Thank you very much for your valuable suggestion. Our response is provided below.

To further demonstrate the effectiveness of RPS, we conducted additional experiments using Pass@1 and the unnormalized difference between Pass@K and Pass@1 to calculate the shared-core gradient weights for each task, while keeping the gradient core selection strategy unchanged. The results are presented in Table 2. We can observe that ProCore with RPS achieves the best performance on half of the tasks and obtains the highest average performance, further demonstrating the effectiveness of the RPS design.

Table 2. Effect of different shared-core weighting strategies on ProCore performance.

MethodsARC1DCountdownCodeIORubikSokobanZebraWordLadderBitwiseAVG
ProCore (Pass@1)41.557.032.020.514.038.541.565.038.8
ProCore (Pass@K-Pass@1)38.059.532.531.59.036.534.573.039.3
ProCore (RPS)34.057.033.538.529.540.536.567.042.1

The experiments cover multiple Qwen3 sizes, but all models belong to the same family. Do the authors expect the observed gradient concentration and the selected hyperparameters to transfer to other model families?

Thank you very much for your valuable suggestions. Our response is provided below.

To further demonstrate the effectiveness of ProCore, we conducted additional experiments on MiniCPM5-2B, an open-source model from a different model family developed by OpenBMB, while keeping all ProCore hyperparameters unchanged. Since Sokoban is relatively challenging and MiniCPM5-2B consistently obtains a score of zero on this task, we conducted experiments on the remaining seven datasets.

We first analyzed the gradient distributions of individual tasks during multi-task training on MiniCPM5-2B. The results are presented in Table 3. As shown, all tasks exhibit pronounced gradient concentration, with the top 1.0% of gradient components by magnitude accounting for approximately 72.9%–78.6% of the total gradient norm. This result demonstrates that gradient concentration is not unique to the Qwen model family but is also observed in MiniCPM5-2B, further supporting the applicability of ProCore across different model families.

Table 3. Percentage of each task's gradient norm retained by the top-k% largest-magnitude gradient components on MiniCPM5-2B.

Task1.0%1.2%1.4%1.6%1.8%2.0%
ARC73.3%74.9%76.3%77.4%78.5%79.4%
Countdown75.3%76.9%78.2%79.4%80.4%81.3%
CodeIO74.5%76.0%77.3%78.4%79.3%80.2%
Rubik's Cube78.6%80.0%81.1%82.2%83.1%83.9%
Zebra Puzzles73.7%75.3%76.6%77.8%78.8%79.8%
Word Ladder73.9%75.4%76.8%77.9%78.9%79.8%
Bitwise72.9%74.4%75.7%76.8%77.8%78.7%

The multi-task experimental results on MiniCPM5-2B are presented in Table 4. We can observe that ProCore achieves more balanced overall performance across the seven tasks. Furthermore, compared with the strongest baseline (FAMO), ProCore achieves an average improvement of 2.5 percentage points across the seven tasks. This further demonstrates that ProCore is effective not only on the Qwen model family but also offers advantages when applied to other model families.

Table 4. Performance comparison of ProCore and baselines across seven tasks on MiniCPM5-2B.

MethodARC1DCountdownCodeIORubikZebraWordLadderBitwiseAverage
Mixed44.064.034.0018.0034.5012.0053.0037.1
GradNorm48.056.534.021.538.59.046.036.2
MGDA38.551.032.518.032.511.514.528.4
PCGrad50.559.532.018.033.013.518.032.1
CAGrad46.551.033.511.535.015.019.530.3
Nash-MTL52.564.034.516.540.012.527.535.4
FAMO49.064.034.032.030.511.538.537.1
SEC57.556.031.518.040.04.512.031.4
DADS53.556.032.022.541.016.017.534.1
MGS45.058.534.518.535.012.016.531.4
ProCore57.564.033.025.532.514.550.039.6
AuthorsOpenReview ↗

Response to Reviewer uRhb—Part3/5

In addition, Modular Gradient Surgery also uses task-specific gradients within transformer modules to address multi-domain RL interference. Could the authors clarify the main differences from MGS and explain whether a direct comparison is feasible?

Thank you very much for your valuable comment. We provide three responses to address this concern.

  1. PCGrad detects gradient conflicts across tasks at the full-model level and mitigates them through gradient projection. MGS extends this approach to individual Transformer modules by computing pairwise inner products between task gradients within each module and projecting gradients that conflict. However, MGS still performs conflict detection and projection using the full gradients within each module. In contrast, ProCore applies additional gradient protection only to a small subset of gradient components. Specifically, we identify gradient cores based on gradient magnitudes within each Transformer block. On top of the original mixed gradient, we directly enhance each task's exclusive gradient cores and apply RPS-based weighting to shared gradient cores, while leaving all non-core gradients unchanged. Therefore, ProCore does not require pairwise gradient conflict detection or gradient projection across tasks.

  2. PCGrad and MGS require retaining the full gradient of each task, which can lead to GPU out-of-memory (OOM) errors. To address this issue, we offload task gradients to CPU memory in our implementation and transfer them back to the GPU only when needed for gradient projection and aggregation. Although this allows us to run these methods without OOM errors, it introduces substantial data-transfer and memory overhead. In our experiments, PCGrad takes approximately 1.67 times as long to run as ProCore. Due to this overhead, we initially included only two representative methods in this category, PCGrad and CAGrad, as baselines.

  3. Furthermore, to provide a more comprehensive evaluation of ProCore, we conducted additional experiments on Qwen3-4B, comparing it with several additional multi-task training methods, including MGDA (NeurIPS 2018), GradNorm (ICML 2018), Nash-MTL (ICML 2022), CGPO (ICLR 2026), and MGS (arXiv 2026). The results are presented in Table 5. Compared with these additional baselines, ProCore achieves the best or tied-best performance on four out of eight tasks and obtains the highest average score, further demonstrating its effectiveness in multi-task training.

    Table 5. Performance comparison of ProCore with additional multi-task optimization methods.

    MethodsARC1DCountdownCodeIORubikSokobanZebraWordLadderBitwiseAVG
    GradNorm41.556.532.029.520.539.037.571.040.9
    MGDA36.552.532.018.014.039.532.082.038.3
    Nash-MTL38.552.531.524.011.540.538.573.538.8
    CGPO51.562.031.532.520.033.537.051.539.9
    MGS40.555.533.521.511.540.537.571.038.9
    ProCore34.057.033.538.529.540.536.567.042.1
Reviewer_goXWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ProCore changes multi-task GRPO by finding the top 1.1% of gradient coordinates for each task and giving them a small extra update. It also weights shared coordinates using a score from pass@1 to pass@8. The authors test 8 reasoning tasks on Qwen3-4B/8B.

Critical strengths

The 8B results are promising. ProCore beats mixed training on 7 out of 8 tasks, averaging 44.0 Vs. 40.6 over three seeds (Table 1), with about 5% extra runtime. It also improves DAPO by 2.9 points and MT-GRPO by 1.9 (Table 5).

Critical weaknesses
  1. Most of the 4B gain comes from Rubik’s Cube (+19.7) and Sokoban (+12.2). Across the other six tasks, the average barely changes (44.6 versus 44.3), and ARC1D drops 3.3 points. The settings also appear to have been chosen using the evaluation examples.
  2. The paper claims to fix gradient cancellation, but does not show that cancellation happens. Figure 6 keeps 12.3–15.4% of each task’s signal, close to the 12.5% expected from averaging eight tasks. Random coordinates already reach 40.5 versus ProCore’s 42.1 (Table 3).
Questions for authors
  1. Were the core ratio, rho and selection strategy (Tables 6 to 8) chosen on the 200 evaluation examples?
  2. What is the plus or minus in Table 1, and why are the Qwen3-1.7B GPQA-Diamond scores in Table 2 below the 25 percent chance level?
4 comments
AuthorsOpenReview ↗

Response to Reviewer goXW—Part1/4

Most of the 4B gain comes from Rubik’s Cube (+19.7) and Sokoban (+12.2). Across the other six tasks, the average barely changes (44.6 versus 44.3), and ARC1D drops 3.3 points.

Thank you very much for your comment. We provide two responses to address this concern.

  1. First, ProCore performs mixed training across eight tasks that exhibit significant differences in input formats, reasoning processes, and answer spaces. There is substantial interference among different tasks. We calculate the proportion of shared-core gradients that conflict with those of other tasks for each of the eight tasks during ProCore training. The results are presented in Table 1. On both Qwen3-4B and Qwen3-8B, more than 70% of the shared-core gradients of each task conflict with those of other tasks, indicating that gradient conflicts among shared cores are prevalent. For two conflicting tasks, an improvement in one task is often accompanied by a decline in the other. Therefore, existing methods (e.g., GradNorm, MGDA, PCGrad, CAGrad, Nash-MTL, FAMO, SEC, DADS, MGS, CGPO, and MOPD) struggle to improve all eight tasks simultaneously. We can also observe from Table 1 of our paper that, on Qwen3-4B, compared with Mixed Training, PCGrad leads to a 10.2 percentage-point drop in Bitwise, SEC leads to drops of 9.5 and 17.2 percentage points in Sokoban and Bitwise, respectively, DADS leads to a 5.2 percentage-point drop in Sokoban, and MOPD leads to a 9 percentage-point drop in Bitwise. In contrast, ProCore improves the performance on Rubik's Cube and Sokoban while largely avoiding performance degradation on the other tasks, resulting in a more balanced overall performance.

    Table 1: Gradient conflict ratios in shared cores across eight tasks.

    ModelARC1DCountdownCodeIORubik's CubeSokobanZebra PuzzlesWord LadderBitwise ArithmeticAvg
    Qwen3-4B73.43%73.21%73.87%73.86%73.75%73.90%74.62%73.73%73.80%
    Qwen3-8B73.09%72.55%73.13%72.99%72.30%73.08%73.71%72.48%72.92%
  2. Second, the performance gains of ProCore do not come solely from Rubik's Cube and Sokoban. ProCore is built upon Mixed Training. Across the eight tasks in our main experiments, compared with Mixed Training, ProCore achieves improvements on 5 tasks with Qwen3-4B and 7 tasks with Qwen3-8B. Furthermore, across the nine tasks in our generalization experiments, compared with Mixed Training, ProCore achieves scores no lower than those of Mixed Training on 8, 7, and 7 tasks with Qwen3-1.7B, Qwen3-4B, and Qwen3-4B-Base, respectively.

AuthorsOpenReview ↗

Response to Reviewer goXW—Part2/4

Were the core ratio, rho and selection strategy (Tables 6 to 8) chosen on the 200 evaluation examples?

We sincerely apologize for the misunderstanding caused by the wording in our paper. We would like to further clarify the hyperparameter selection procedure. Specifically, we randomly held out 5% of the original training data from each task as an independent validation set (500 examples per task) and conducted hyperparameter selection experiments on Qwen3-4B using this validation set to determine the gradient core selection ratio (κ\kappa) and the gradient core enhancement strength (ρ\rho). The corresponding experimental results on the validation set are reported in Tables 2 and 3. Based on these validation results, we ultimately selected κ=1.1%\kappa=1.1\% and ρ=0.1\rho=0.1. It is worth noting that Table 7 in Appendix A.2 and Table 8 in Appendix A.3 of the paper report the test-set performance of models trained with different hyperparameter configurations. These results are only intended to demonstrate model performance under different hyperparameter configurations and were not involved in the hyperparameter selection process. We will add the details of the hyperparameter selection experiments to the revised paper.

Table 2: Validation performance of ProCore with different gradient core selection ratios (κ\kappa).

MethodARC1DCountdownCodeIORubikSokobanZebraWordLadderBitwiseAVG
ProCore (κ=1.0%\kappa=1.0\%)38.051.033.033.511.539.039.065.038.8
ProCore (κ=1.1%\kappa=1.1\%)30.555.533.039.527.040.538.566.041.3
ProCore (κ=1.2%\kappa=1.2\%)35.557.029.539.019.539.039.060.539.9

Table 3: Validation performance of ProCore with different core enhancement coefficients (ρ\rho).

MethodARC1DCountdownCodeIORubikSokobanZebraWordLadderBitwiseAVG
ProCore (ρ=0.05\rho=0.05)38.052.030.534.512.538.036.071.039.1
ProCore (ρ=0.1\rho=0.1)30.555.533.039.527.040.538.566.041.3
ProCore (ρ=0.2\rho=0.2)35.556.034.039.016.537.033.066.539.7
AuthorsOpenReview ↗

Response to Reviewer goXW—Part3/4

The paper claims to fix gradient cancellation, but does not show that cancellation happens. Figure 6 keeps 12.3–15.4% of each task’s signal, close to the 12.5% expected from averaging eight tasks.

Thank you for your valuable comment. First, we would like to clarify that the gradient attenuation discussed in our paper consists of two aspects: (1) Gradient scaling induced by the mixing coefficients: each task's gradient is scaled by its corresponding mixing coefficient. For example, when eight tasks are equally weighted, each task's gradient contribution is scaled by a factor of one-eighth; (2) Gradient cancellation across different tasks: when gradients from different tasks point in opposite directions along the same parameter dimensions, the corresponding gradient signals are further weakened. Figure 6 illustrates the combined effects of these two factors, rather than cancellation caused solely by conflicting gradient directions. On the other hand, compared with Mixed Training, ProCore improves the gradient core retention ratio across all eight tasks, with an average improvement of 2.52 percentage points, indicating that it better preserves each task's core update signals in the aggregated gradient.

Random coordinates already reach 40.5 versus ProCore’s 42.1 (Table 3).

Thank you for your valuable comment. We would like to clarify that although w/o MCS in Table 3 adopts random selection of gradient cores, it still retains the Dynamic Task Protection (DTP) mechanism. Specifically, for shared cores, protection weights are still dynamically assigned according to the RPS scores of the corresponding tasks, giving higher weights to tasks with greater remaining learning potential. For exclusive gradient cores, direct enhancement is still applied. Therefore, this variant can still dynamically protect task gradients through RPS, which explains why it achieves relatively good performance. Nevertheless, compared with full ProCore, randomly selecting gradient cores leads to average performance drops of 1.6 and 3.3 percentage points on Qwen3-4B and Qwen3-8B, respectively, indicating that identifying gradient cores based on gradient magnitudes remains important for improving overall performance.

What is the plus or minus in Table 1?

Thank you for your valuable comment. The plus or minus (±) in Table 1 of our paper represents the standard deviation of the experimental results across three runs with different random seeds. Each result is reported as the mean plus or minus the standard deviation.

Why are the Qwen3-1.7B GPQA-Diamond scores in Table 2 below the 25 percent chance level?

Thank you for your valuable comment. When setting up GPQA-Diamond, we referred to the approach used in CGPO (ICLR 2026). Specifically, we provide four options (A, B, C, and D) in the prompt and explicitly instruct the model to end its response with Final Answer: <letter>. We then use rule-based parsing to extract the model's final choice from its response. If the extraction fails or the selected option is incorrect, the response is counted as incorrect and receives a score of 0. Therefore, the model's score is affected by both answer correctness and adherence to the required output format.

AuthorsOpenReview ↗

Response to Reviewer goXW—Part4/4

Supplementary Experiment 1

To further validate the effectiveness of ProCore, we conduct additional comparative experiments on Qwen3-4B with several multi-task training methods, including MGDA (NeurIPS 2018), GradNorm (ICML 2018), Nash-MTL (ICML 2022), CGPO (ICLR 2026), and MGS (arXiv 2026). The experimental results are presented in Table 4. Compared with these additional baselines, ProCore achieves the best performance on half of the tasks and obtains the highest average performance, further validating the effectiveness of ProCore in multi-task training.

Table 4: Performance comparison of ProCore with additional multi-task optimization methods.

MethodARC1DCountdownCodeIORubikSokobanZebraWordLadderBitwiseAVG
GradNorm41.556.532.029.520.539.037.571.040.9
MGDA36.552.532.018.014.039.532.082.038.3
Nash-MTL38.552.531.524.011.540.538.573.538.8
CGPO51.562.031.532.520.033.537.051.539.9
MGS40.555.533.521.511.540.537.571.038.9
ProCore34.057.033.538.529.540.536.567.042.1

Supplementary Experiment 2

To further demonstrate the effectiveness of ProCore, we conducted additional experiments on MiniCPM5-2B, an open-source model from a different model family developed by OpenBMB, while keeping all ProCore hyperparameters unchanged. Since Sokoban is relatively challenging and MiniCPM5-2B consistently obtains a score of zero on this task, we conducted experiments on the remaining seven datasets.

We first analyzed the gradient distributions of individual tasks during multi-task training on MiniCPM5-2B. The results are presented in Table 5. As shown, all tasks exhibit pronounced gradient concentration, with the top 1.0% of gradient components by magnitude accounting for approximately 72.9%–78.6% of the total gradient norm. This result demonstrates that gradient concentration is not unique to the Qwen model family but is also observed in MiniCPM5-2B, further supporting the applicability of ProCore across different model families.

Table 5: Percentage of each task's gradient norm retained by the top-k% largest-magnitude gradient components on MiniCPM5-2B.

Task1.0%1.2%1.4%1.6%1.8%2.0%
ARC73.3%74.9%76.3%77.4%78.5%79.4%
Countdown75.3%76.9%78.2%79.4%80.4%81.3%
CodeIO74.5%76.0%77.3%78.4%79.3%80.2%
Rubik's Cube78.6%80.0%81.1%82.2%83.1%83.9%
Zebra Puzzles73.7%75.3%76.6%77.8%78.8%79.8%
Word Ladder73.9%75.4%76.8%77.9%78.9%79.8%
Bitwise72.9%74.4%75.7%76.8%77.8%78.7%

The multi-task experimental results on MiniCPM5-2B are presented in Table 6. We can observe that ProCore achieves more balanced overall performance across the seven tasks. Furthermore, compared with the strongest baseline (FAMO), ProCore achieves an average improvement of 2.5 percentage points across the seven tasks. This further demonstrates that ProCore is effective not only on the Qwen model family but also offers advantages when applied to other model families.

Table 6. Performance comparison of ProCore and baselines across seven tasks on MiniCPM5-2B.

MethodARC1DCountdownCodeIORubikZebraWordLadderBitwiseAverage
Mixed44.064.034.0018.0034.5012.0053.0037.1
GradNorm48.056.534.021.538.59.046.036.2
MGDA38.551.032.518.032.511.514.528.4
PCGrad50.559.532.018.033.013.518.032.1
CAGrad46.551.033.511.535.015.019.530.3
Nash-MTL52.564.034.516.540.012.527.535.4
FAMO49.064.034.032.030.511.538.537.1
SEC57.556.031.518.040.04.512.031.4
DADS53.556.032.022.541.016.017.534.1
MGS45.058.534.518.535.012.016.531.4
ProCore57.564.033.025.532.514.550.039.6
Reviewer_2QkvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ProCore, a plug-and-play gradient method for multi-task reinforcement learning with verifiable rewards. At each update, the method computes task-specific gradients, selects the largest-magnitude coordinates within each Transformer block as task gradient cores, and partitions them into exclusive and shared cores according to coordinate overlap. Exclusive cores receive direct enhancement, while shared cores are combined using weights derived from a Remaining Potential Score (RPS). RPS is computed from a smoothed pass@1 statistic and the maximum pass@8 observed over recent training steps.

Experiments use Qwen3 models on eight procedurally generated reasoning tasks and on a second collection of mathematical, coding, and scientific benchmarks. The paper reports improvements over mixed training and several gradient or RLVR baselines, with additional ablations and integrations with DAPO and MT-GRPO. The main conceptual claim is that protecting concentrated task-specific gradient coordinates can reduce destructive interference and preserve task learning potential.

Critical strengths

Simple and implementation-friendly mechanism. The exclusive/shared core distinction can be inserted into GRPO-style training without changing the model architecture or requiring a separate inference module.

More than a single benchmark comparison. The paper evaluates multiple model scales, external benchmarks, ablations, and combinations with DAPO and MT-GRPO. This provides some evidence that the heuristic is not entirely tied to one configuration.

Critical weaknesses

The central RPS and update rule are insufficiently justified. Pass@8 is the maximum success among eight samples from the current policy, not an upper bound on attainable performance. Taking a maximum over recent steps makes it optimistic and noisy. The update adds ρgi\rho g_i on top of the ordinary mixed gradient; for an exclusive core, this effectively gives a task an additional weight beyond the nominal mixture coefficient. Shared-core weights are determined by RPS rather than by a clearly derived objective. There is no descent argument, update normalization, convergence analysis, or evidence that the gains cannot be reproduced by simple task reweighting or increasing the update norm.

The empirical evidence is narrow and the compute claim is questionable. Main experiments run for only 120 steps on heavily engineered reasoning tasks, with limited seeds and small evaluation sets. The gains are concentrated in Rubik and Sokoban, while ARC1D declines and several tasks are nearly unchanged. The paper claims only about 5% extra wall-clock time despite computing eight task gradients and maintaining additional core statistics, but it does not provide enough forward/backward counts, peak-memory, rollout-token, or batch-organization details to make this credible. Direct baselines such as MGS, CGPO, Nash-MTL, MGDA, and GradNorm are also missing or insufficiently evaluated.

Questions for authors

Please provide a held-out calibration study showing whether RPS predicts improvement over the next 10–20 updates better than current reward, advantage, pass@1 trend, or a simple fixed task weight. In addition, include matched controls with the same update norm: core enhancement without RPS, RPS weighting applied to all coordinates, and scalar per-task reweighting. This is necessary to determine whether the gains come from RPS-based protection rather than implicit task reweighting or a larger effective learning rate.

Please report exact forward/backward counts, rollout tokens, peak memory, communication cost, and wall-clock time for Mixed, ProCore, PCGrad/CAGrad, DAPO, and MT-GRPO under matched token and compute budgets. Given that ProCore computes separate gradients for eight tasks, how can its total cost increase by only approximately 5%?

Please rerun the main ablations and external benchmarks over multiple random seeds with means, standard deviations, confidence intervals, and significance tests, using a validation split to select κ\kappa, ρ\rho, and the core-selection hyperparameters. Please also report per-task changes and Pareto fronts rather than relying primarily on an unweighted average.

13 comments
AuthorsOpenReview ↗

Response to Reviewer 2Qkv—Part 1/12

The central RPS and update rule are insufficiently justified.

Thank you for your valuable comment. To help you better understand RPS, we elaborate below on the motivation behind its design and its update rule.

1. Motivation behind RPS. In multi-task RLVR, different tasks exhibit different learning speeds. We aim to prioritize the protection of tasks that learn more slowly. However, it is difficult to accurately characterize the learning progress of each task. Directly using rewards to measure learning progress may overlook tasks that improve slowly but still have substantial room for improvement. Previous studies have found that in single-task RLVR, Pass@1 and Pass@K gradually converge as training proceeds [1][2][3]. We observe the same phenomenon for individual tasks in multi-task RLVR. Motivated by this observation, we use the Pass@K at each training step as a reference upper bound on the model's capability at that step. We then measure the task's learning progress based on the gap between its current Pass@1 and this reference upper bound. During training, both Pass@1 and Pass@K change continuously at each step, reflecting the evolving learning progress of each task.

2. Specific Design and Update Rule of RPS. In our implementation, we use the average Pass@1 over the three most recent training steps as the current Pass@1, and the maximum Pass@K over the five most recent training steps as the current Pass@K, which serves as the reference upper bound for the current step. Since different tasks have different initial performance levels and reference upper bounds, directly using the gap between Pass@1 and Pass@K makes it difficult to measure the relative learning progress across tasks. It may also excessively favor certain tasks and thus affect the balance of multi-task learning. Therefore, we use the difference between the current reference upper bound and the initial Pass@1 (i.e., the performance before training) as the denominator, and the difference between the current reference upper bound and the current Pass@1 as the numerator. This gives us the normalized learning progress metric, RPS. At initialization, the RPS of all tasks is set to 1. As shown in Figure 4 of our paper, the RPS of each task gradually approaches 0 as training proceeds, but the rates of decrease differ substantially across tasks, further reflecting the differences in their learning speeds.

Pass@8 is the maximum success among eight samples from the current policy, not an upper bound on attainable performance.

Pass@K can serve as a reference upper bound for the model's capability. Previous studies have also used Pass@K as a reference for estimating the upper bound of model capability [1][4][5][6]. In our experiments, we set K to the group size of GRPO to avoid the additional cost of extra rollouts. We have also conducted extensive experiments to demonstrate the effectiveness of this choice. Furthermore, while keeping the group size fixed at 8, we conduct additional experiments on Qwen3-4B using Pass@12 and Pass@16 as reference upper bounds. The results are presented in Table 1. As K increases, Pass@K provides a more accurate reference upper bound, and the overall performance of ProCore gradually improves. However, the performance gains become progressively smaller rather than increasing indefinitely.

Table 1: Effect of different Pass@K settings on ProCore performance.

MethodARC1DCountdownCodeIORubikSokobanZebraWordLadderBitwiseAVG
ProCore(Pass@8)34.057.033.538.529.540.536.567.042.1
ProCore(Pass@12)42.063.034.531.518.542.538.573.543.0
ProCore(Pass@16)40.056.034.047.025.038.035.072.043.4

References:

[1] Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (Yue et al., NeurIPS 2025.)

[2] The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Reward (Li et al., ICLR 2026.)

[3] Understanding Diversity Collapse in RLVR via the Lens of Overtraining (Yuan et al., arXiv 2026.)

[4] Reasoning with Exploration: An Entropy Perspective (Cheng et al., AAAI 2026.)

[5] Beyond Pass@1: Self-Play with Variational Problem Synthesis Sustains RLVR (Liang et al., ICLR 2026.)

[6] Weight Ensembling Improves Reasoning in Language Models (Dang et al., COLM 2025.)

AuthorsOpenReview ↗

Response to Reviewer 2Qkv—Part 2/12

Taking a maximum over recent steps makes it optimistic and noisy.

Thank you very much for your comment. ProCore uses the maximum Pass@K over several recent training steps as the reference upper bound to preserve the best performance that the model has recently achieved and use it to estimate how much room for improvement remains on the current task. If smoothing methods such as the moving average were used instead, the reference upper bound would also decrease when a task's performance drops. This could lead to prematurely concluding that the task is approaching its reference upper bound, thereby reducing its weight. Moreover, since Pass@1 and Pass@8 at each training step are calculated over the entire training batch, the influence of an occasional high score from an individual sample on the overall estimate is limited. Finally, we conduct additional experiments on Qwen3-4B using the average Pass@8 over several recent training steps as the reference upper bound for the current step. The results are presented in Table 2. ProCore using the maximum Pass@8 over recent training steps as the reference upper bound achieves performance no lower than its average-based counterpart on six out of the eight tasks, while achieving better overall performance. These results further demonstrate the effectiveness of our original design.

Table 2: Comparison of different strategies for estimating the reference upper bound.

MethodARC1DCountdownCodeIORubikSokobanZebraWordLadderBitwiseAVG
ProCore(Avg)41.054.033.520.021.540.536.571.039.8
ProCore(Max)34.057.033.538.529.540.536.567.042.1

The update adds (\rho g_i) on top of the ordinary mixed gradient; for an exclusive core, this effectively gives a task an additional weight beyond the nominal mixture coefficient. Shared-core weights are determined by RPS rather than by a clearly derived objective.

Thank you very much for your comment. We would like to clarify that ProCore introduces an additional core-gradient enhancement term on top of the standard mixed gradient, rather than uniformly reweighting the entire task gradient. For the exclusive cores of task i, this enhancement term is ρgi\rho g_i. For the shared cores of task i, the enhancement term is ρwigi\rho w_i g_i, where wiw_i is determined based on the RPS of task i. This is because an exclusive core is selected by only one task, indicating that only this task identifies the corresponding gradient component as important. In contrast, a shared core is selected by multiple tasks. Therefore, its enhancement weight needs to be dynamically allocated based on RPS to provide greater protection to tasks with slower learning progress. Furthermore, we acknowledge that the weights for shared cores are heuristic and designed based on RPS. However, dynamically allocating weights based on training feedback is a common practice in multi-task RLVR. For example, SEC, DADS, and FAMO dynamically adjust task sampling ratios or optimization weights based on advantage statistics, historical rewards, and the relative decrease in task losses, respectively. PAC [1] further adaptively allocates training budgets across tasks by combining advantage statistics with recent reward improvements. In addition, the ablation study in Table 3 of our paper also verifies the effectiveness of RPS-based dynamic protection weights compared with fixed weights.

References:

[1] PAC: Progress-Augmented Advantage Curriculum for Multi-Task Reinforcement Learning of LLMs (Yu et al., EMNLP 2026).

AuthorsOpenReview ↗

Response to Reviewer 2Qkv—Part 3/12

There is no descent argument, update normalization, convergence analysis, or evidence that the gains cannot be reproduced by simple task reweighting or increasing the update norm.

Thank you for your valuable comment. We provide three responses to address this concern.

  1. First, the primary focus of this paper is to mitigate the attenuation of each task's core gradients during multi-task RLVR and empirically validate the effectiveness of the core-gradient protection mechanism, rather than provide a theoretical analysis. Regarding update magnitude control, ProCore does not amplify gradients without constraints. Specifically, according to Eqs. (10) and (11), the core enhancement term is computed as a weighted sum of the corresponding task gradients using normalized Softmax weights and is controlled by the enhancement coefficient ρ\rho (0.1 by default). Therefore, its coordinate-wise magnitude has a well-defined upper bound. Moreover, before the optimizer update, we apply global norm clipping to the final aggregated gradient to ensure that its norm does not exceed a predefined threshold, thereby further controlling the magnitude of the gradient used for the update. Finally, the training reward curves in Figures 8 and 9 of Appendix C.4 exhibit an overall stable convergence trend, further providing empirical support for the training stability of ProCore.

  2. Second, unlike conventional task-reweighting methods, ProCore goes beyond assigning a single weight to each task's entire gradient. Conventional task reweighting methods typically apply a uniform weight to all gradient dimensions of the same task. In contrast, ProCore first identifies task-specific gradient cores based on gradient magnitudes and partitions them into exclusive and shared cores according to their dimensional overlap across tasks. Based on this partitioning, ProCore directly enhances exclusive cores and dynamically allocates protection weights to shared cores based on RPS, while keeping non-core dimensions unchanged. Furthermore, Table 1 of our paper already compares ProCore with these task reweighting methods, including FAMO, SEC, DADS, and DynaSample. To provide a more detailed comparison between ProCore and these methods, we extract the experimental results on Qwen3-4B and present them in Table 3. As shown, ProCore achieves the best results on half of the tasks and obtains the highest overall average score among these methods. These results further demonstrate that the performance gains brought by ProCore cannot be achieved through simple task reweighting alone.

    Table 3: Comparison of ProCore with four different task reweighting methods.

    MethodARC1DCountdownCodeIORubikSokobanZebraWordLadderBitwiseAVG
    DADS42.8±3.055.2±1.028.7±0.533.8±3.710.8±1.740.2±2.035.2±1.070.0±0.439.6±0.5
    SEC39.0±5.052.3±1.127.5±0.028.3±4.66.5±0.742.5±0.735.0±0.049.5±0.735.1±1.2
    DynaSample34.2±0.352.8±2.632.7±0.330.5±4.312.0±2.840.8±2.136.3±2.566.5±0.938.2±1.6
    FAMO38.2±2.853.5±1.133.3±0.225.8±0.819.5±0.441.5±2.335.5±1.563.2±0.838.8±0.6
    ProCore33.7±2.555.3±1.433.2±2.439.5±0.828.2±1.042.2±2.437.2±0.966.2±0.641.9±0.4
  3. Finally, we further examine whether ProCore's performance gains can be attributed solely to an increased update norm. To this end, we introduce an additional baseline, MT-LR, by increasing the learning rate of Mixed Training to (1+ρ)(1+\rho) times its original value, thereby increasing the update norm. The experimental results are presented in Table 4. ProCore outperforms MT-LR on almost all tasks, achieving average scores of 42.1% and 44.5% on Qwen3-4B and Qwen3-8B, respectively, compared with 37.7% and 41.4% for MT-LR. These results demonstrate that simply increasing the update norm is insufficient to reproduce ProCore's performance gains, further highlighting the importance of selectively protecting task-specific core gradients.

    Table 4: Performance comparison between ProCore and MT-LR.

    ModelMethodARC1DCountdownCodeIORubikSokobanZebraWordLadderBitwiseAVG
    Qwen3-4BMT-LR38.052.033.523.020.034.035.565.537.7
    Qwen3-4BProCore34.057.033.538.529.540.536.567.042.1
    Qwen3-8BMT-LR49.553.033.025.020.046.034.070.541.4
    Qwen3-8BProCore53.056.036.035.020.544.039.072.544.5

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.