- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
In multi-task reinforcement learning, learning multiple tasks with a shared model can lead to interference from easier tasks, hindering the learning of harder tasks. To address this issue, this paper proposes Cross-level Progress Recovery (CPR), a method that combines inter-task dropout with intra-task prioritization. CPR first introduces the expectile temporal-difference (XTD) error to identify transitions that remain underestimated by the critic. Specifically, it trains an XTD model to estimate the upper expectile of the TD error. At the inter-task level, CPR selectively drops tasks that exhibit both higher returns and reduced XTD errors to mitigate inter-task interference. At the intra-task level, CPR prioritizes high-XTD transitions to extract more informative learning signals from harder tasks. Experimental results on MT-10 and MT-45 demonstrate the effectiveness of the proposed CPR method.
Critical strengths
-
The paper is clearly written and easy to follow. Figure 1 effectively illustrates the motivation behind CPR by showing how return and XTD are used to assess learning progress across single-task and multi-task settings, as well as tasks of varying difficulty. It also provides a clear overview of the proposed method and its implementation details.
-
Although expectile regression is not new and has been widely adopted for value estimation in offline RL, its application to identifying underestimated samples in multi-task RL through expectile temporal-difference error is fresh. The experimental results further demonstrate the effectiveness of this approach. CPR is evaluated against eight baselines across five random seeds, demonstrating its superior performance. The paper also provides per-task learning curves in the appendix for a more comprehensive evaluation. Furthermore, it conducts detailed ablation studies on the proposed inter-task dropout and intra-task prioritization components, validating their effectiveness.
Critical weaknesses
-
In multi-task RL, low returns and large Q-value estimation errors generally indicate poor policy performance and inaccurate value estimation, respectively, but do not necessarily imply that a task is inherently difficult to learn. In my view, both inter-task dropout and intra-task prioritization primarily emphasize tasks or samples that appear to be insufficiently learned. Although CPR may provide harder tasks with more training opportunities, it does not necessarily resolve interference between tasks of different difficulty levels, as the underlying conflicts between their learning objectives may persist. As shown in Figure D.2 in the appendix, CPR performs worse than the baselines on some hard tasks.
-
For inter-task dropout, CPR needs to rank task returns and estimate XTD errors for each task. As the number of tasks increases, these operations may introduce additional computational overhead and reduce scalability.
Questions for authors
-
Since policy improvement and policy evaluation influence each other, does fixing the policies to random policies reflect the practical Q-value estimation error shown in Figure 2?
-
Is the output of the expectile function a scalar or a vector? How to estimate the XTD error for each task?
2 comments
W2 (Computational Overhead and Scalability)
We thank the reviewer for the comments on computational overhead and scalability as the number of tasks increases. We expanded the existing cost analysis in Appendix G to distinguish network computation from task-level bookkeeping. CPR trains a single shared XTD estimator on the same minibatches used for critic updates and tracks per-task statistics using EMAs, avoiding full-buffer reevaluation. It does not require a separate estimator for each task. Updating these statistics costs , while sorting-based return ranking costs . These operations are additional to the actor, critic, and shared XTD network updates, rather than duplicating the training procedure for every task.
Comparing throughput on MT10 and MT50, a fivefold increase in the number of tasks corresponded to only an approximately 15% increase in the estimated time to process the same number of environment transitions per task. Under the observed execution configurations, the estimated training-time increase was therefore substantially smaller than the fivefold increase in task count. This supports the practical scalability of the implementation over the evaluated range and suggests that task ranking and XTD estimation do not introduce a dominant task-count bottleneck. It does not imply constant-cost scaling or a universal advantage over other implementations. We have added these explanations and comparison results to Appendix G.
Q2 (XTD Estimation and Task-Level Aggregation)
The XTD estimator outputs a single scalar representing the conditional -expectile of signed TD errors. A single shared network takes the state and task representation as inputs and learns this estimate by minimizing the expectile regression loss in Equation (3), as clarified in Section 4.1.
Once again, we thank the reviewer for the careful review and feedback. We hope the additional explanations and analyses adequately address the concerns and questions raised.
We thank the reviewer for the comments and questions on the definition of hard tasks and CPR’s effectiveness, computational overhead and scalability, the analysis using fixed policies, and the XTD estimation procedure. Below, we respond to each point and describe the corresponding revisions to the manuscript.
Q1 (Purpose of Fixed-Policy Q-Bias Analysis)
We thank the reviewer for raising a question about the fixed-policy Q-bias analysis, which is presented in Figure 3 rather than Figure 2. Since policy improvement and evaluation influence each other, we deliberately fix the policies of Push, Pick-Place, and Peg-Insert-Side to random policies to eliminate the influence of policy improvement, while allowing the remaining tasks to learn. This controlled setting isolates the effect of learning other tasks on shared-critic value estimation. Since the fixed policies have unchanged true Q-values, their estimation errors are expected to decrease as critic learning progresses.
To further validate this interpretation, we added a single-task control experiment using the same fixed policies and matched per-task critic updates. Figure 3 shows that Q bias decreases and remains low under single-task training but increases substantially under multi-task training, with final mean values of 4.73 and 88.54, respectively. This contrast provides stronger evidence that learning other tasks interferes with shared-critic value estimation, even without policy improvement for the analyzed tasks. Importantly, the experiment is intended to isolate inter-task interference rather than reproduce the full policy-improvement process. We clarified the purpose of this controlled experiment in Section 4.2 and added the single-task comparison to Figure 3, further supporting our interpretation of inter-task interference.
W1 (Hard-Task Definition and Effectiveness of CPR)
We thank the reviewer for the comments on the definition of hard tasks and CPR's effectiveness. We agree that low returns or large Q-value estimation errors do not necessarily indicate intrinsic task difficulty. In our work, hard tasks refer to tasks whose learning progress is hindered under multi-task training, particularly by interference from other tasks, rather than tasks that are inherently difficult to learn. As illustrated in Figure 1, Pick-Place progresses under single-task learning but stagnates in the multi-task setting. We clarified this distinction in Section 1.
Importantly, CPR does not merely provide more training opportunities to hard tasks but introduces XTD as an explicit criterion for determining when and where to intervene in learning. ITD mitigates interference from sufficiently learned tasks through the shared critic, while ITP emphasizes states with large positive TD corrections to improve value learning in hard tasks. We strengthened this motivation through Proposition 1 in Section 4.1. Specifically, Let δ denote the signed TD error, μ=E[δ] its mean, and ηq its exact q-expectile (XTD), where q∈(0.5,1). Proposition 1 establishes that
ηq−μ=(2q−1)E[∣δ−ηq∣]=1−q2q−1E[(δ−ηq)+]≥0,
where (x)+=max(x,0). This establishes XTD's upper-tail emphasis, providing a theoretical basis for highlighting large positive TD corrections that may be obscured by averaging. The Q-bias analysis in Figure 3 further shows that substantial Q bias arises under joint training despite fixed policies. These results support our approach to mitigating interference-induced value-estimation errors rather than claiming to eliminate all conflicts between task objectives.
We further expanded the MT50 task-wise analysis in Appendix E to examine CPR's effectiveness and limitations. CPR performs well on hammer and stick-push, while ITP and ITD contribute notably to pick-out-of-hole and peg-unplug-side, respectively. In contrast, some baselines outperform CPR on handle-pull variants, and all methods struggle with assembly and disassemble, suggesting limitations beyond inter-task interference that may benefit from complementary approaches such as network scaling. Nevertheless, CPR achieves the highest Worst-20 success rate among the compared reset-free methods. Additional MT50 ablations also confirm the contributions of both ITD and ITP, as shown in Table R.2 in our response to Reviewer MMt3. These findings support CPR's effectiveness in recovering learning progress across challenging tasks while acknowledging its remaining limitations.