- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates multi-task RLVR at the element level. Specifically, it first argues that RLVR task gradients are rather highly concentrated. This motivates this paper to first identify the task gradient cores and then partition them into shared and exclusive ones. Thereafter, it employs an RPS score, which is estimated from the ratio between Pass@1 and Pass@5 of past rollout scores, to reweight task gradients. Extensive experiments conducted on multiple datasets across models have demonstrated the effectiveness of the proposed method.
Critical strengths
-
The motivation of this paper is rather clear and easy to follow.
-
The proposed solution is technically sound and quite simple.
-
This paper is well-written, with a clear structure.
Critical weaknesses
-
All designs are quite straightforward and intuitive. From my side, they might lack novelty on the technology level. And I believe the proposed solution is not strong corrective with the high concentration observation; they appearly could be applied to more general scenarios. I also attribute the performance gains to the element-level operations rather than to what the authors claimed.
-
Since ProCore operates at the element-level, a complexity analysis and comparison is required.
-
In Figure 3, how are exclusive cores enhanced? According to my understanding, only shared cores are weighted; why are exclusive cores not specially weighted?
-
The choice of is too intuitive, and it could be easily uneffective when both Pass@1 and Pass@5 are pool during the early training stage.
Questions for authors
Please refer to the Critical Weaknesses. Besides, no limitations are explicitly claimed in this paper.
5 comments
Response to Reviewer bsWf—Part2/4
Since ProCore operates at the element-level, a complexity analysis and comparison is required.
Thank you very much for your valuable suggestion. Our response is provided below.
-
Theoretical Complexity. Let N denote the number of tasks and D denote the number of model parameters. We focus on the time and space complexity of gradient processing and aggregation. ProCore identifies gradient cores for each task and applies weighted enhancement to the selected approximately 1.1% of gradient components, with a time complexity of O(ND). In our implementation, although ProCore sequentially accumulates the base gradients from individual tasks, it retains only the indices and values of each task's gradient cores, without simultaneously storing the full gradients of all tasks. Its space complexity is therefore O(D+0.011ND). In addition, RPS reuses reward statistics already collected during training and requires no additional sampling. In comparison, PCGrad performs pairwise gradient projections across the full model parameter space, with a time complexity of O(N²D) for gradient processing. CAGrad constructs an inter-task gradient inner-product matrix and further optimizes the aggregation weights. The complexity of all four methods is summarized in Table 2.
Table 2. Time and space complexity of gradient processing and aggregation
Method Time Complexity Space Complexity Mixed Training PCGrad CAGrad + Weight Optimization Overhead ProCore -
Actual Runtime Overhead. We conducted experiments using Qwen3-4B on two NVIDIA A100 80GB GPUs, running Mixed Training, PCGrad, CAGrad, and ProCore under identical training configurations. We measured the average total runtime per step over the first three complete training steps. As shown in Table 3, ProCore increases the per-step training time by only 2.53% compared with Mixed Training. Its runtime is also substantially lower than that of PCGrad and CAGrad, demonstrating that ProCore can process and aggregate task gradients with relatively little additional runtime overhead.
Table 3. Runtime comparison of different methods on Qwen3-4B.
Method Training Step Time (s/step) Mixed Training 555.90 PCGrad 951.97 CAGrad 787.33 ProCore 569.95
Response to Reviewer bsWf—Part3/4
In Figure 3, how are exclusive cores enhanced? According to my understanding, only shared cores are weighted; why are exclusive cores not specially weighted?
Thank you for your comment. First, we would like to clarify that "Direct Enhancement" in Figure 3 refers to directly enhancing exclusive cores, rather than leaving them unweighted. Specifically, for a gradient component selected as a core only by task , ProCore directly adds times the corresponding task-specific core gradient to the original mixed gradient, i.e., . Here, is the core enhancement coefficient, which is set to 0.1 in our main experiments. We do not apply additional RPS weighting to exclusive cores because an exclusive core is selected by only one task, so there is no need to allocate protection weights among multiple tasks. According to Eq. (10), when , the normalized weight is naturally . In contrast, for shared cores, multiple tasks select the same component as a core, so their relative protection weights need to be dynamically determined based on their RPS values. Therefore, "Direct Enhancement" and "RPS-based Weight" in Figure 3 correspond to direct enhancement of exclusive cores and RPS-weighted enhancement of shared cores, respectively. Both operations enhance the core gradients on top of the original mixed gradient.
The choice of is too intuitive, and it could be easily uneffective when both Pass@1 and Pass@5 are pool during the early training stage.
Thank you for your comment. We provide three responses to address this concern.
-
First, we would like to clarify that our method uses Pass@8 rather than Pass@5. Specifically, at each training step, we generate 8 responses for each question and compute the mean Pass@1 and Pass@8 across all questions in that step to obtain the Pass@1 and Pass@8 for the current step, respectively. We set to the maximum Pass@8 over the most recent 5 training steps as the empirical performance ceiling, where 5 denotes the history window size. Furthermore, An unusually high score on an individual sample has limited influence on the overall estimate, since Pass@1 and Pass@8 are computed over all training data in each step.
-
Second, when calculating RPS, the numerator is the difference between the empirical performance ceiling and the average Pass@1 over the three most recent training steps, while the denominator is the difference between the same ceiling and the initial Pass@1 before training. Therefore, even if both Pass@1 and Pass@8 are low during early training, RPS will remain valid as long as the model's performance during training exceeds its performance before training. We also clip RPS to ensure that its value remains within the range of [0, 1]. Figure 4 in our paper shows the evolution of RPS during training, demonstrating that no apparent numerical instability occurs.
-
Finally, previous studies have used Pass@K as a reference for the upper bound of model capability [1][2][3][4]. In designing ProCore, we use the maximum Pass@K over recent training steps as the reference upper bound, with the aim of preserving the best performance the model has recently achieved and using it to estimate how much room for improvement remains on the current task. If we instead use a moving average or other smoothing strategies, the estimated upper bound may decrease when task performance drops, potentially leading us to prematurely conclude that a task has reached its upper bound and consequently reducing its weight. In addition, we conducted experiments on Qwen3-4B using the average Pass@8 over recent training steps as the upper bound for the current step. The results are presented in Table 4. ProCore using the maximum Pass@8 over recent training steps as the upper bound achieves scores no lower than those of the average-based variant on six tasks, while also achieving better overall performance, further validating the effectiveness of our original design.
Table 4: Comparison of different strategies for estimating the reference upper bound.
Method ARC1D Countdown CodeIO Rubik Sokoban Zebra WordLadder Bitwise AVG ProCore (Avg) 41.0 54.0 33.5 20.0 21.5 40.5 36.5 71.0 39.8 ProCore (Max) 34.0 57.0 33.5 38.5 29.5 40.5 36.5 67.0 42.1
References:
[1] Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (Yue et al., NeurIPS 2025.)
[2] Reasoning with Exploration: An Entropy Perspective (Cheng et al., AAAI 2026.)
[3] Beyond Pass@1: Self-Play with Variational Problem Synthesis Sustains RLVR (Liang et al., ICLR 2026.)
[4] Weight Ensembling Improves Reasoning in Language Models (Dang et al., COLM 2025.)
Response to Reviewer bsWf—Part1/4
Thank you very much for your valuable comment. We would like to further clarify the contributions of ProCore and its connection to the observed gradient concentration.
Technical novelty. ProCore introduces a mechanism for identifying and protecting task-specific gradient cores. Beyond the original mixed gradient, ProCore retains the high-magnitude gradient components of each task, distinguishes exclusive and shared cores based on cross-task overlap at each dimension, and uses RPS to determine the protection weights for shared cores. Consequently, even within the same task, different dimensions receive different treatments depending on their core membership. This fine-grained, dimension-level protection mechanism cannot be achieved solely through task-level sampling ratio or weight adjustments (e.g., FAMO, SEC, DADS, DynaSample, and MT-GRPO). Furthermore, unlike multi-task optimization methods such as PCGrad and CAGrad, which require storing and processing the full gradient of each task, ProCore retains only the indices and values of approximately 1.1% of the gradient components for each task after gradient computation, for subsequent protection and enhancement, without needing to store all task gradients simultaneously. This substantially reduces memory overhead and the cost of subsequent gradient processing. Taking PCGrad as an example, to avoid OOM issues, we offload each task's full gradient to CPU memory during training and transfer it back to the GPU as needed for gradient projection and aggregation. This incurs substantial memory and data-transfer overhead, making PCGrad approximately 1.67 times slower to train than ProCore.
Gradient concentration directly determines the protection targets and scope of ProCore. We find that the top 1% of gradient components by magnitude account for an average of 87.3% of each task's gradient norm. This indicates that the primary gradient signals of each task are concentrated in a small subset of dimensions. Based on this observation, ProCore does not uniformly modify the entire gradient. Instead, it restricts additional gradient protection to a small subset of core dimensions. Specifically, within each Transformer block, we select approximately 1.1% of the gradient components with the largest magnitudes as gradient cores. Building on the original mixed gradient, we directly enhance exclusive cores and apply RPS-weighted enhancement to shared cores, while leaving non-core dimensions unchanged. Therefore, the gradient concentration observation directly guides ProCore's core selection and selective enhancement strategy, enabling it to specifically protect the primary gradient signals of each task.
The performance gains are not solely attributable to element-wise operations. In Table 3 of the paper, both ablation variants retain the element-wise enhancement operation but modify either the core selection strategy or the computation of shared-core gradient weights. Replacing magnitude-based core selection with random selection at the same ratio reduces the average scores by 1.6 and 3.3 percentage points on Qwen3-4B and Qwen3-8B, respectively. Replacing dynamic RPS-based weights with equal weights leads to further reductions of 0.9 and 1.6 percentage points, respectively. Moreover, we conducted additional experiments using Pass@1 and the Pass@K–Pass@1 gap to compute the weights for shared-core gradients, while retaining the element-wise operations. The results are presented in Table 1. ProCore using RPS achieves the highest average performance. Overall, these results demonstrate that ProCore's performance gains cannot be attributed solely to element-wise operations.
Table 1. Effect of different shared-core weighting strategies on ProCore performance.