acceptodds
Under review as a conference paper at ICLR 2027

CPR: Reviving Hard-Task Learning in Multi-Task Reinforcement Learning

Abstract

Multi-task reinforcement learning (MTRL) aims to learn multiple tasks with a shared model, yet hard tasks often make limited progress when trained alongside easier ones. We identify weak temporal-difference (TD) learning signals as a key characteristic of this failure and introduce expectile temporal-difference error (XTD), an upper expectile of signed TD errors that reveals informative positive value corrections obscured by averaging. Building on XTD, we propose Cross-level Progress Recovery (CPR), which revives hard-task learning through complementary inter-task and intra-task mechanisms. Inter-task dropout uses relative return and declining task-level XTD to identify sufficiently learned tasks and mitigate shared-critic over-specialization, while intra-task prioritization emphasizes high-XTD states to strengthen informative learning signals within hard tasks. Experiments on Meta-World show that CPR substantially improves overall performance, particularly on hard tasks, without relying on parameter resets. Trajectory and component analyses show that XTD highlights critical states and support the complementary roles of both mechanisms in reducing critic interference and improving hard-task learning.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Multi-Task Reinforcement Learning, Task Imbalance, Value Estimation Bias, Adaptive Regularization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_k7pHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

In multi-task reinforcement learning, learning multiple tasks with a shared model can lead to interference from easier tasks, hindering the learning of harder tasks. To address this issue, this paper proposes Cross-level Progress Recovery (CPR), a method that combines inter-task dropout with intra-task prioritization. CPR first introduces the expectile temporal-difference (XTD) error to identify transitions that remain underestimated by the critic. Specifically, it trains an XTD model to estimate the upper expectile of the TD error. At the inter-task level, CPR selectively drops tasks that exhibit both higher returns and reduced XTD errors to mitigate inter-task interference. At the intra-task level, CPR prioritizes high-XTD transitions to extract more informative learning signals from harder tasks. Experimental results on MT-10 and MT-45 demonstrate the effectiveness of the proposed CPR method.

Critical strengths
  1. The paper is clearly written and easy to follow. Figure 1 effectively illustrates the motivation behind CPR by showing how return and XTD are used to assess learning progress across single-task and multi-task settings, as well as tasks of varying difficulty. It also provides a clear overview of the proposed method and its implementation details.

  2. Although expectile regression is not new and has been widely adopted for value estimation in offline RL, its application to identifying underestimated samples in multi-task RL through expectile temporal-difference error is fresh. The experimental results further demonstrate the effectiveness of this approach. CPR is evaluated against eight baselines across five random seeds, demonstrating its superior performance. The paper also provides per-task learning curves in the appendix for a more comprehensive evaluation. Furthermore, it conducts detailed ablation studies on the proposed inter-task dropout and intra-task prioritization components, validating their effectiveness.

Critical weaknesses
  1. In multi-task RL, low returns and large Q-value estimation errors generally indicate poor policy performance and inaccurate value estimation, respectively, but do not necessarily imply that a task is inherently difficult to learn. In my view, both inter-task dropout and intra-task prioritization primarily emphasize tasks or samples that appear to be insufficiently learned. Although CPR may provide harder tasks with more training opportunities, it does not necessarily resolve interference between tasks of different difficulty levels, as the underlying conflicts between their learning objectives may persist. As shown in Figure D.2 in the appendix, CPR performs worse than the baselines on some hard tasks.

  2. For inter-task dropout, CPR needs to rank task returns and estimate XTD errors for each task. As the number of tasks increases, these operations may introduce additional computational overhead and reduce scalability.

Questions for authors
  1. Since policy improvement and policy evaluation influence each other, does fixing the policies to random policies reflect the practical Q-value estimation error shown in Figure 2?

  2. Is the output of the expectile function a scalar or a vector? How to estimate the XTD error for each task?

2 comments

We thank the reviewer for the comments and questions on the definition of hard tasks and CPR’s effectiveness, computational overhead and scalability, the analysis using fixed policies, and the XTD estimation procedure. Below, we respond to each point and describe the corresponding revisions to the manuscript.

Q1 (Purpose of Fixed-Policy Q-Bias Analysis)

We thank the reviewer for raising a question about the fixed-policy Q-bias analysis, which is presented in Figure 3 rather than Figure 2. Since policy improvement and evaluation influence each other, we deliberately fix the policies of Push, Pick-Place, and Peg-Insert-Side to random policies to eliminate the influence of policy improvement, while allowing the remaining tasks to learn. This controlled setting isolates the effect of learning other tasks on shared-critic value estimation. Since the fixed policies have unchanged true Q-values, their estimation errors are expected to decrease as critic learning progresses.

To further validate this interpretation, we added a single-task control experiment using the same fixed policies and matched per-task critic updates. Figure 3 shows that Q bias decreases and remains low under single-task training but increases substantially under multi-task training, with final mean values of 4.73 and 88.54, respectively. This contrast provides stronger evidence that learning other tasks interferes with shared-critic value estimation, even without policy improvement for the analyzed tasks. Importantly, the experiment is intended to isolate inter-task interference rather than reproduce the full policy-improvement process. We clarified the purpose of this controlled experiment in Section 4.2 and added the single-task comparison to Figure 3, further supporting our interpretation of inter-task interference.

W1 (Hard-Task Definition and Effectiveness of CPR)

We thank the reviewer for the comments on the definition of hard tasks and CPR's effectiveness. We agree that low returns or large Q-value estimation errors do not necessarily indicate intrinsic task difficulty. In our work, hard tasks refer to tasks whose learning progress is hindered under multi-task training, particularly by interference from other tasks, rather than tasks that are inherently difficult to learn. As illustrated in Figure 1, Pick-Place progresses under single-task learning but stagnates in the multi-task setting. We clarified this distinction in Section 1.

Importantly, CPR does not merely provide more training opportunities to hard tasks but introduces XTD as an explicit criterion for determining when and where to intervene in learning. ITD mitigates interference from sufficiently learned tasks through the shared critic, while ITP emphasizes states with large positive TD corrections to improve value learning in hard tasks. We strengthened this motivation through Proposition 1 in Section 4.1. Specifically, Let δ\delta denote the signed TD error, μ=E[δ]\mu=\mathbb E[\delta] its mean, and ηq\eta^q its exact qq-expectile (XTD), where q∈(0.5,1)q\in(0.5,1). Proposition 1 establishes that

ηq−μ=(2q−1)E[∣δ−ηq∣]=2q−11−qE[(δ−ηq)+]≥0,\displaystyle \eta^q-\mu = (2q-1)\mathbb E[\lvert \delta-\eta^q\rvert] = \frac{2q-1}{1-q}\mathbb E[(\delta-\eta^q)_{+}] \geq 0,

where (x)+=max⁡(x,0)(x)_{+}=\max(x,0). This establishes XTD's upper-tail emphasis, providing a theoretical basis for highlighting large positive TD corrections that may be obscured by averaging. The Q-bias analysis in Figure 3 further shows that substantial Q bias arises under joint training despite fixed policies. These results support our approach to mitigating interference-induced value-estimation errors rather than claiming to eliminate all conflicts between task objectives.

We further expanded the MT50 task-wise analysis in Appendix E to examine CPR's effectiveness and limitations. CPR performs well on hammer and stick-push, while ITP and ITD contribute notably to pick-out-of-hole and peg-unplug-side, respectively. In contrast, some baselines outperform CPR on handle-pull variants, and all methods struggle with assembly and disassemble, suggesting limitations beyond inter-task interference that may benefit from complementary approaches such as network scaling. Nevertheless, CPR achieves the highest Worst-20 success rate among the compared reset-free methods. Additional MT50 ablations also confirm the contributions of both ITD and ITP, as shown in Table R.2 in our response to Reviewer MMt3. These findings support CPR's effectiveness in recovering learning progress across challenging tasks while acknowledging its remaining limitations.

W2 (Computational Overhead and Scalability)

We thank the reviewer for the comments on computational overhead and scalability as the number of tasks increases. We expanded the existing cost analysis in Appendix G to distinguish network computation from task-level bookkeeping. CPR trains a single shared XTD estimator on the same minibatches used for critic updates and tracks per-task statistics using EMAs, avoiding full-buffer reevaluation. It does not require a separate estimator for each task. Updating these statistics costs O(N)O(N), while sorting-based return ranking costs O(Nlog⁡N)O(N\log N). These operations are additional to the actor, critic, and shared XTD network updates, rather than duplicating the training procedure for every task.

Comparing throughput on MT10 and MT50, a fivefold increase in the number of tasks corresponded to only an approximately 15% increase in the estimated time to process the same number of environment transitions per task. Under the observed execution configurations, the estimated training-time increase was therefore substantially smaller than the fivefold increase in task count. This supports the practical scalability of the implementation over the evaluated range and suggests that task ranking and XTD estimation do not introduce a dominant task-count bottleneck. It does not imply constant-cost scaling or a universal advantage over other implementations. We have added these explanations and comparison results to Appendix G.

Q2 (XTD Estimation and Task-Level Aggregation)

The XTD estimator ηψq(s,zi)\eta_\psi^q(s,z_i) outputs a single scalar representing the conditional qq-expectile of signed TD errors. A single shared network takes the state and task representation as inputs and learns this estimate by minimizing the expectile regression loss in Equation (3), as clarified in Section 4.1.

Once again, we thank the reviewer for the careful review and feedback. We hope the additional explanations and analyses adequately address the concerns and questions raised.

Reviewer_MMt3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Cross-level Progress Recovery (CPR) to address learning stagnation on hard tasks in MTRL. The authors introduce Expectile Temporal-Difference Error (XTD) to capture informative positive TD corrections that may be obscured by averaging. Based on XTD, CPR combines inter-task dropout to mitigate shared-critic over-specialization toward sufficiently learned tasks and intra-task prioritization to emphasize informative states within hard tasks. Experiments on Meta-World MT10 and MT50 demonstrate substantial performance improvements over existing baselines without relying on parameter resets. Ablation studies and additional analyses further investigate the effectiveness of the proposed components.

Critical strengths
  1. The paper addresses the important problem of learning stagnation on hard tasks in multi-task reinforcement learning, and the proposed method is well motivated.
  2. The method provides a valuable unified signal for assessing learning progress and guiding training, while also being computationally efficient.
  3. The paper demonstrates substantial performance improvements, supported by relatively comprehensive experimental analyses.
Critical weaknesses
  1. The paper interpret large positive TD errors as signals of insufficient value learning by the critic, but high XTD values may also arise from stochastic transitions, variance in Bellman targets, or irreducible estimation noise, and such signals may persist even after the critic has converged. The authors should further investigate the quantitative relationship between XTD and subsequent critic error reduction or actual learning progress to better support their core interpretation.

  2. In Section 4.2, it is unclear whether the increase in Q bias is indeed caused by interference from easier tasks? Furthermore, the paper measures Q bias using the absolute difference between rollout returns and critic Q-values, whereas the SAC critic objective includes entropy regularization. If the rollout returns refer to standard environment returns, these two quantities are not strictly consistent. The authors should clarify this issue and provide stronger evidence supporting their explanation of inter-task interference.

  3. Although the paper demonstrates that XTD outperforms Mean TD, it lacks comparisons with more straightforward alternatives, such as Positive TD Error, Absolute TD Error, or standard Prioritized Experience Replay. Therefore, it remains unclear whether the performance improvements stem from the unique ability of Upper Expectile to capture sparse positive corrections or simply from emphasizing samples with large TD errors. Given that XTD is a central contribution of the paper, these comparisons are important for establishing its necessity.

  4. CPR achieves a Worst-10 Success Rate of only 0.5% on MT50, indicating that the most challenging tasks remain largely unsolved. The authors should further analyze what distinguishes these tasks from those whose learning progress is successfully recovered, and identify the conditions under which CPR can or cannot revive learning. Such analysis should also be conducted on MT50, rather than limiting the ablation studies primarily to MT10.

Questions for authors

See Weakness

4 comments

We thank the reviewer for the thoughtful comments on the relationship between XTD and learning progress, the analysis of inter-task interference, comparisons with alternative prioritization methods, and limitations on hard tasks. Below, we respond to each point and describe the corresponding revisions to the manuscript.

W1/W2 (Interpretation of XTD and Evidence of Inter-Task Interference)

W1. We thank the reviewer for highlighting the relationship between XTD's interpretation and actual learning progress. We agree that large TD errors can arise from stochastic transitions, Bellman-target variance, or irreducible noise and may persist after convergence. We therefore do not interpret high XTD as a guarantee of insufficient learning or reducible error. Instead, XTD aims to highlight states with large positive value-correction signals that may be obscured by the mean signed TD error. To clarify this interpretation, we strengthened both the theoretical characterization of XTD and the empirical evidence for its usefulness.

Most importantly, we have added Proposition 1 to Section 4.1, with its proof in Appendix A. Specifically, let δ\delta denote the signed TD error, μ=E[δ]\mu=\mathbb{E}[\delta] its mean, and ηq\eta^q its qq-expectile (XTD), where q∈(0.5,1)q\in(0.5,1). Proposition 1 establishes that

ηq−μ=(2q−1)E[∣δ−ηq∣]=2q−11−qE[(δ−ηq)+]≥0,\displaystyle \eta^q-\mu = (2q-1)\mathbb{E}[\lvert \delta-\eta^q\rvert] = \frac{2q-1}{1-q}\mathbb{E}[(\delta-\eta^q)_{+}] \geq 0,

where (x)+=max⁡(x,0)(x)_{+}=\max(x,0). The first equality shows that XTD is no smaller than mean TD, with the gap increasing as the expectile level qq increases for nonconstant TD-error distributions. The second equality shows that the gap is proportional to the expected upper-tail excess at fixed qq. Under the stated upper-tail enlargement, XTD increases even when mean TD remains unchanged. Consequently, when underestimation relative to bootstrapped targets produces large positive TD errors, XTD can highlight them more strongly than mean TD. This characterizes upper-tail emphasis rather than guaranteeing that XTD separates correctable underestimation from noise.

W2. The Q-bias analysis in Section 4.2 and Figure 3 examines a possible source of such value-estimation errors: interference from learning other tasks through the shared critic. In the original experiment, we fixed the policies of Push, Pick-Place, and Peg-Insert-Side to random policies while allowing the remaining tasks to learn and continuing critic updates on all tasks. The true values of the three fixed policies therefore remain unchanged. However, the original multi-task-only experiment did not establish how much estimation error would arise without joint training. We therefore added a single-task control experiment using the same fixed policies and matched per-task critic updates. Figure 3 shows that Q bias decreases and remains low under single-task training but increases substantially under multi-task training, with final mean values of 4.73 and 88.54, respectively. This contrast provides stronger evidence that learning other tasks interferes with shared-critic value estimation, beyond the error observed when each policy is evaluated separately. We also clarified in Section 4.2 that the original measurement compares a separate reward-only Q-function, now denoted by Q′Q', with discounted rollout returns under the same fixed policy, excluding entropy regularization to keep the two quantities consistent.

Finally, returning to W1's question about subsequent critic-error reduction, we used the same fixed-random-policy setting to compare critic learning with ITP (q=0.9q=0.9) alone against uniform weighting, excluding all other CPR components. As shown in Figure 6, ITP reduced the mean top-10% TD error across the three tasks faster than uniform weighting, reaching 3.31 after 2K training steps, compared to 4.90 with uniform weighting. Thus, Proposition 1 explains which part of the error distribution XTD emphasizes, Figure 3 provides evidence of value-estimation errors arising from joint training, and Figure 6 demonstrates that XTD-guided updates can accelerate the reduction of large TD errors in the evaluated setting. Together, these additions strengthen the theoretical and empirical basis for XTD-guided critic learning. The theoretical analysis and proof are provided in Section 4.1 and Appendix A, the revised Q-bias analysis in Section 4.2 and Figure 3, and the additional TD-error reduction experiment in Section 5.3 and Figure 6.

W3 (Comparison with Alternative Prioritization Methods)

We thank the reviewer for suggesting comparisons with other TD-error-based prioritization methods. We agree that comparison with Mean TD alone is insufficient to establish the benefit of XTD over simpler alternatives. Following the suggestion, we replaced XTD-based ITP with Positive TD and Absolute TD prioritization, and separately replaced ITP with prioritized experience replay (PER), while keeping ITD and all other training settings unchanged. We evaluated these variants alongside the existing Mean TD and ITP-removal ablations, as reported in Table R.1.

On MT10, Mean TD (85.3%), Absolute TD (85.8%), and PER (71.9%) perform substantially below CPR. Positive TD performs better at 92.3%, suggesting that emphasizing positive corrections associated with underestimation relative to the TD target is more useful than emphasizing error magnitude alone. Nevertheless, XTD-based CPR achieves the highest success rate of 97.0%. The distinction is that Positive TD directly prioritizes observed positive errors from individual transitions, whereas XTD estimates the upper expectile of signed TD errors conditioned on the state and task. Thus, even when an individual transition has a small observed TD error, XTD can assign greater priority if its state exhibits substantial positive correction signals. These comparisons strengthen the empirical evidence that state-conditioned upper-tail information provides benefits beyond simply increasing the weight of large observed TD errors.

We have incorporated these comparison results into Section 5.4, Table 3 and clarified the distinctions between XTD-based ITP and other TD-error-based prioritization methods.

Table R.1. Additional comparisons with TD-error-based prioritization methods on MT10.

VariantSuccess Rate
MT-SAC73.7 ± 2.2
CPR w/o Intra-Task Prioritization81.5 ± 9.0
CPR w/ PER71.9 ± 4.8
CPR w/ Mean TD85.3 ± 2.6
CPR w/ Absolute TD85.8 ± 5.0
CPR w/ Positive TD92.3 ± 1.7
CPR97.0 ± 0.1

W4 (Learning Limitations and Ablation Analysis on MT50)

Thank you for suggesting additional MT50 ablations and task-wise analysis. We acknowledge that the low Worst-10 success rate indicates that some challenging tasks remain unsolved. To better understand these limitations, we expanded the task-wise analysis in Appendix E. CPR outperforms the baselines on hammer and stick-push, which involve tool-mediated manipulation, while ITP contributes notably to pick-out-of-hole and ITD to peg-unplug-side. In contrast, some baselines perform better on handle-pull and handle-pull-side, and all methods struggle with assembly and disassemble, which require precise manipulation. These results suggest that CPR is effective when learning can benefit from mitigating shared-critic interference and emphasizing remaining value corrections, but may be insufficient for tasks with difficulties beyond these mechanisms. Such limitations could potentially be addressed through complementary approaches, including improved exploration or network scaling. Nevertheless, CPR achieves the highest Worst-20 success rate among the compared reset-free methods on MT50, indicating broader improvements across challenging tasks despite the remaining limitations.

We also extended the ITD and ITP removal ablations to MT50, with results presented below and in Table F.1 of Appendix F.5. Removing either component reduces performance relative to full CPR, while both ablated variants still substantially outperform MT-SAC. These results confirm that ITD and ITP contribute to performance improvements on MT50, extending their effectiveness beyond MT10. Together, the additional analyses clarify both the complementary contributions of CPR's components and the limitations that remain on particularly challenging tasks.

Table R.2. Component ablations on MT50.

MethodAverage Success Rate (%)
MT-SAC49.4 ± 1.7
CPR w/o Inter-Task Dropout67.5 ± 2.5
CPR w/o Intra-Task Prioritization64.6 ± 1.1
CPR72.7 ± 3.3

Once again, we thank the reviewer for the careful review and feedback. We hope the additional analyses, experiments, and explanations adequately address the concerns raised.

Thank you for your detailed response and additional experimental results, which have addressed most of my concerns. I acknowledge this is a good paper, but I do not believe it quite reaches the level of rating 4. Therefore, I have decided to maintain my original positive score to support the paper.

We sincerely thank the reviewer for the thoughtful reassessment and positive assessment of our work. The reviewer's constructive feedback throughout the discussion has been valuable in helping us clarify our contributions and strengthen the manuscript. We greatly appreciate the time and careful consideration devoted to our paper.

Reviewer_hwhyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes that weak temporal difference signals is a limiting factor in multi-task reinforcement learning (MTRL). To overcome this they propose leveraging expectile regression, inter-task dropout, and intra-task prioritization to overcome this issue. Results on the Meta-World benchmark seem to indicate that the proposed method, CPR, outperforms all baselines.

Critical strengths

The paper was well written and easy to follow.

The results are fairly convincing.

Critical weaknesses
  1. Lack of appropriate baselines. McLean et al 2025a showed that recent methods (Soft Modularization, MOORE, and PaCo) are mostly unneeded, and scaling the neural network is the reason behind the performance improvements. So comparing the CPR method to these other methods doesn't make a ton of sense to me. I think addressing this somewhere in the text is important

2.. There should be other MTRL baselines that do have the spirit of McLean et al 2025a, such as Nauman et al 2025 which showed that a combination of distributional critics, n-step returns, and a few other small tricks made MTRL more sample efficient than learning tasks in single policies. I think addressing this gap is extremely important. The comparison between expectile regression and distributional RL would be very valuable.

Questions for authors
  1. If you remove all of the components of CPR down to a base neural network, is it just a MT-SAC algorithm training a vanilla neural network?

I was unsure where to state this: Isn't the Meta-World citation now McLean et al 2025b? It looks like you used the baselines that McLean et al 2025b released but didn't cite them.

Citations

Michal Nauman, Marek Cygan, Carmelo Sferrazza, Aviral Kumar, Pieter Abbeel. Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learners. NeurIPS 2025.

Reginald McLean, Evangelos Chatzaroulas, J.K. Terry, Isaac Woungang, Nariman Farsad, Pablo Samuel Castro. Multi-Task Reinforcement Learning Enables Parameter Scaling. RLC 2025a.

Reginald McLean, Evangelos Chatzaroulas, Luc McCutcheon, Frank Röder, Tianhe Yu, Zhanpeng He, K.R. Zentner, Ryan Julian, J KTerry, Isaac Woungang, Nariman Farsad, Pablo Samuel Castro. Meta-World+: An Improved, Standardized, RL Benchmark. NeurIPS 2025b.

4 comments

We thank the reviewer for the comments on baseline comparisons and model scaling, as well as the questions regarding CPR’s relationship to MT-SAC and the citation of prior work. Below, we respond to each point and describe the corresponding revisions to the manuscript.

W1/W2 (Baseline Comparisons across Network Scales)

We thank the reviewer for the comments on baseline selection and model scaling and for suggesting additional comparisons. We agree that recent work on network scaling and advanced architectures, including McLean et al. [3] and BRC [4], should be discussed and evaluated. We have expanded Section 2 to distinguish two complementary directions: increasing network capacity or developing more expressive architectures, and improving learning frameworks within a given network capacity. CPR follows the latter direction, using XTD as an explicit criterion for determining when and where to regulate and prioritize learning. This direction remains important for robots and embedded systems, where memory and computational constraints can limit network size and larger models increase training costs. Our baseline selection is also not limited to older methods: Soft Modules, MOORE, and PaCo remain part of the comparisons in recent studies such as ARS [1] and HyMTRL [2], both of which are included in our evaluation. These comparisons assess improvements in learning methods under controlled capacity, rather than assuming that such methods replace the benefits of scaling.

Nevertheless, we agree that these baselines alone do not address the requested comparison with recent scaling approaches. Following the feedforward scaling setup of McLean et al. [3], we used a three-layer feedforward architecture and evaluated MT-MH-SAC, BRC*, and CPR on MT10 at critic widths of 400 and 2048. A direct capacity-matched comparison with the original BRC is difficult because its BroNet-based critic uses approximately 135M parameters, compared with approximately 2M for CPR. We therefore constructed BRC*, a variant using the same MLP-based backbone as CPR, and kept parameter counts similar across methods at each width. This comparison evaluates BRC's algorithmic design with a matched backbone, not the full original BroNet configuration. Table R.1 is also provided as Table F.2 in Appendix F.6. The results are as follows.

Table R.1. Success rates across network widths on MT10.

Method4002048
MT-MH-SAC78.3 ± 8.072.4 ± 0.3
BRC*86.7 ± 0.596.7 ± 1.0
CPR97.0 ± 0.198.2 ± 0.7

Within this matched setup, increasing network width produces little change in MT-MH-SAC, whereas BRC* and CPR improve with scale, suggesting that increasing network capacity alone is insufficient and that effective learning methods remain important. CPR further outperforms BRC* at both widths, with a larger gap at width 400, highlighting the benefit of XTD-guided training under limited model capacity. Notably, BRC models return distributions through categorical critics, whereas XTD estimates upper expectiles of signed TD errors to guide learning, so the comparison evaluates their respective training frameworks rather than isolating expectile regression against distributional RL. We clarified these distinctions in Section 2 and added the detailed experimental setup and results to Appendix F.6.

References

[1] Cho et al., "ARS: Adaptive reward scaling for multi-task reinforcement learning", ICML 2025.

[2] He et al., "HyMTRL: A Hybrid Multi-Task Reinforcement Learning Framework via Phased Policy Evolution", ICML 2026.

[3] McLean et al., "Multi-Task Reinforcement Learning Enables Parameter Scaling", RLC 2025.

[4] Nauman et al., "Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learners", NeurIPS 2025.

Q1 (Relationship between CPR and MT-SAC)

We thank the reviewer for the question about CPR’s architecture. Yes. Removing the proposed ITD and ITP, together with the auxiliary return prioritization and gradient normalization, yields exactly basic MT-SAC with the same shared, task-conditioned actor and critic networks. CPR’s core contribution lies in improving the training method rather than modifying the network architecture or increasing model size.

Q2 (Benchmark Citation and Attribution)

Thank you for pointing out the missing citation to Meta-World+. We addressed this omission by adding the Meta-World+ [5] citation in Section 5.1 and Appendices C.1 and C.2, together with explicit descriptions of the environment version and the source of the baseline code.

References

[5] McLean et al., "Meta-World+: An Improved, Standardized, RL Benchmark", NeurIPS 2025.

Once again, we thank the reviewer for the careful review and feedback. We hope the additional comparisons and explanations adequately address the concerns raised.

Thank you for these extra experiments. The 2048 MT-MH-SAC result is extremely low compared to the results of McLean et al 2025. I would recommend that the authors look into that a bit more.

Other than that, my comments have been addressed effectively.

Thank you for your careful feedback and for pointing out this discrepancy. Although we used the original code provided by McLean et al. for our MT-MH-SAC experiments, we found that our experiments relied on an earlier version of the implementation, before commit https://github.com/rainx0r/mtrl/commit/f5c121f734823a5fd66642a7fadc52917143184d, which addressed issues related to Q-value clipping and truncation handling. We repeated the comparison using the updated MT-MH-SAC implementation with these issues resolved, as shown in Table R.1.1, and found that its performance remains similar at width 400 but improves substantially at width 2048, consistent with the scaling results reported by McLean et al. Nevertheless, CPR retains higher success rates than both the updated MT-MH-SAC and BRC* at matched widths, supporting the complementary value of XTD-guided training in improving learning within a given network capacity, even as that capacity increases.

We have incorporated the updated MT-MH-SAC results into Table 1 and Table F.3 of the revised manuscript. We appreciate the reviewer's careful feedback, which helped us identify and resolve this issue, and hope that the updated comparison adequately addresses the remaining concern.

Table R.1.1. Success rates across network widths on MT10, including MT-MH-SAC results from the previous and updated implementations.

Method4002048
MT-MH-SAC (Previous Commit)78.3 ± 8.072.4 ± 0.3
MT-MH-SAC (Updated Commit)81.0 ± 7.787.3 ± 2.3
BRC*86.7 ± 0.296.7 ± 1.0
CPR (Ours)97.0 ± 0.198.2 ± 0.7

Comments on the paper

2 comments

We sincerely thank the Area Chair and all reviewers for their careful review and constructive feedback. During the rebuttal period, we strengthened CPR's theoretical foundations, clarified its contributions, and conducted additional experiments addressing concerns about inter-task interference, prioritization strategies, and scalability. The main revisions, highlighted in blue in the revised manuscript, are summarized below.

(i) Theoretical and Empirical Clarification of XTD (Reviewers hwhy, MMt3, k7pH; Sections 2, 4.1, and 5.3, Appendix A)

A central concern was whether CPR offers meaningful improvements beyond network scaling and conventional training adjustments, and whether XTD captures informative learning signals rather than noise. We clarified in Section 2 that CPR focuses on improving learning under a given network capacity, with XTD providing an explicit criterion for determining when and where learning should be regulated or prioritized. Our most significant revision is the addition of Proposition 1 in Section 4.1, with its proof in Appendix A, which formally establishes XTD's upper-tail emphasis relative to mean TD. The analysis shows that the gap increases with the expectile level qq and reflects upper-tail TD errors, providing a theoretical basis for highlighting large positive value corrections that may be obscured by averaging. We further supported this property with an experiment in Section 5.3 showing that XTD-based ITP accelerates the reduction of large TD errors. Together, these revisions strengthen the theoretical and empirical justification for XTD-guided learning in hard tasks.

(ii) Additional Evidence of Inter-Task Interference (Reviewers MMt3, k7pH; Section 4.2, Figure 3)

Reviewers MMt3 and k7pH questioned the interpretation and validity of the fixed-random-policy Q-bias experiment. To strengthen the evidence for inter-task interference, we added a controlled single-task comparison with matched critic updates. As shown in Figure 3, Q bias decreases under single-task training but increases substantially under multi-task training, even though the policies remain fixed and their true values do not change. This provides stronger evidence that learning other tasks can interfere with shared-critic value estimation. We also clarified the experimental purpose and the use of a separate reward-only Q-function in Section 4.2.

(iii) Additional TD-Based Prioritization Comparisons and MT50 Ablations (Reviewers MMt3, k7pH; Section 5.4, Appendices E and F.5)

Following the reviewers' requests for stronger ablation evidence, we expanded the evaluation of ITP by comparing alternative TD-based prioritization methods and examining CPR's individual components. Additional comparisons with Positive TD, Absolute TD, and prioritized experience replay (PER) in Section 5.4 show that XTD-based CPR achieves the highest success rate on MT10, supporting the benefit of state-conditioned upper-expectile prioritization over conventional alternatives. We also extended the ITD and ITP removal ablations to MT50 in Appendix F.5 and expanded the task-wise analysis in Appendix E, further examining their complementary contributions and identifying remaining limitations on challenging tasks.

(iv) Network-Scaling Comparisons and Computational Scalability (Reviewers hwhy, k7pH; Appendices F.6 and G)

Reviewer hwhy requested comparisons with recent network-scaling approaches, while k7pH raised concerns about computational overhead as the number of tasks increases. In response, we added comparisons with MT-MH-SAC and BRC under matched network capacities in Appendix F.6, examining the effectiveness of CPR's training mechanisms independently of model scaling. These results highlight the importance of effective training methods, particularly under limited network capacity. We also analyzed the computational complexity of XTD estimation and task-level statistics in Appendix G, finding a manageable increase in processing time when moving from MT10 to MT50. Together, these analyses support the practical value of CPR under constrained model capacity and its potential scalability.

Beyond these major revisions, we clarified the definition of hard tasks, CPR's relationship to MT-SAC, and the XTD estimation procedure, while improving experimental descriptions, citations, and baseline attribution. Detailed responses to each reviewer comment are provided below. We hope these revisions adequately address the concerns raised and clarify CPR's contributions.

Dear Reviewers,

We appreciate the time and effort you have devoted to evaluating our work. Your feedback has helped us strengthen the manuscript through theoretical analysis, clarifications, and additional experiments.

With the discussion period approaching its conclusion, we would welcome any remaining questions or concerns regarding our responses and revisions. We would be happy to address them.

Thank you again for your consideration.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.