- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies when single-policy or multi-policy approaches are more suitable for non-stationary reinforcement learning. The authors divide the environment into several phases and assume that each phase is stationary. Based on this setting, they define transient and quasi-stationary regimes and introduce quantities such as handoff cost and quasi-stationary gain to analyze the trade-offs between the two approaches. The main focus of the paper is not simply whether multi-policy performs better, but under what conditions and why one approach is preferred over the other. The authors support their analysis with experiments in three environments where phase information is observable.
Critical strengths
-
The paper provides a useful quantitative framework for comparing single-policy and multi-policy approaches in non-stationary RL. By distinguishing between transient and quasi-stationary regimes and defining quantities such as handoff cost, the authors help explain under what conditions each approach may be preferable, rather than simply comparing their performance.
-
The paper also provides intuitive explanations for how the analytical quantities are estimated and how they relate to the experimental results. For example, in Section 5.1 (HAPS), the authors explain how they estimate the transient length and use their assumptions to predict a crossover around T = 200. This makes the analysis easier to follow and helps connect the theoretical formulation to the observed results.
Critical weaknesses
-
The paper states in the Introduction that the framework covers both periodic and fixed non-periodic phase structures. However, all experiments are conducted in periodic environments. Since non-periodic settings are also included in the scope of the paper, I think additional validation is needed to support the generality of the analysis.
-
Several aspects discussed in this paper, such as policy specialization and handoff costs, have already been studied in previous work across different application domains, including robotic manipulation. Although the paper brings these findings together into a quantitative framework, it is unclear whether this provides a substantial new contribution beyond combining existing observations.
-
The comparisons between single-policy, multi-head, and multi-policy approaches raise concerns about differences in parameter counts. Previous work, such as Chen et al. (2025), has already examined the trade-off between parameter count and performance in multi-policy RL. Although this paper includes capacity and depth ablations, it is still unclear whether the performance differences come from the policy architecture itself or from differences in model capacity.
-
The analysis considers the behavioral cost of policy handoff but does not account for policy activation latency, which is the time needed for the next policy to produce its first action after switching. If this additional cost is considered, single-policy approaches might be preferable to multi-policy approaches in some settings, especially when switching occurs frequently. Since the paper focuses on when each architecture is preferable, I think this cost should also be considered in the analysis.
Questions for authors
- The paper mentions that the relative ranking of policy architectures may change as the sample budget increases. Could the authors provide a more intuitive explanation of why this might happen, or support this claim with additional experimental results? This would help clarify how the sample budget affects the relative performance of different architectures.
2 comments
Response to Reviewer Mvz8 (2/2)
W3 In our work, we have mostly compared policies under equal learning budget, and we agree that this may not be enough for a fair policy comparison, since model capacity is another factor that plays a role in architecture selection. However, the two factors, i.e., model capacity and learning budget, are interdependent. Therefore, it is difficult to study both of them together; one can study one factor keeping the other fixed.
For example, under a fixed learning budget, one can test if increasing the model capacity improves the performance. Also, for a given model capacity, one can check if a higher learning budget improves the performance. Given a particular learning budget, one can go for an architecture that achieves similar performance with lower model capacity.
In our work, we have increased the model capacity of the single shared policy by increasing depth and width to see if the increased model capacity improves the performance. Our observation is that increasing the shared network’s width or depth raises its active parameter count well above that of one specialist but still under the total of all the specialists, yet none of the tested variants closes the gap in the returns between shared and specialized policies. This argues against active-network size alone explaining the result. Even increasing the depth of the single shared policy above 6 seems to reduce the average reward per step for a single policy. However, this does not establish a capacity-independent architectural effect: the specialist ensemble still has substantially greater total stored capacity, and changing depth also changes optimization difficulty.
To clarify our point, we highlight some of the results. For the default HAP actor--critic models, the results from Appendix A.4.1, Table 7, give us:
| Architecture | Hidden layers | Stored / active parameters | Reward/step |
|---|---|---|---|
| Single+clock | 0.154M / 0.154M | 27.06 | |
| Single+clock, wider | 0.571M / 0.571M | 24.27 | |
| Single+clock, deeper | 0.418M / 0.418M | 29.51 | |
| Single+clock, deeper | 0.681M / 0.681M | 29.04 | |
| Single+clock, deeper | 0.944M / 0.944M | 23.30 | |
| Multihead | shared + heads | 0.246M / 0.153M | 32.76 |
| Multipolicy | 20 separate | 2.862M / 0.143M | 42.34 |
W4 Thank you for identifying a cost that our handoff analysis did not include. To test the policy activation latency between different phases of the multipolicy case, we added an experiment with Phase Pendulum () and the results of Multi policy+clock along with their advantage with respect to the Single policy+clock are presented in the table below (also in Appendix A.4.2 Table 15). Note that Single+clock acts immediately with the correct phase information. For Multi policy+clock, after each boundary, the new specialist supplies no action for steps; during that gap, the actor either holds the previous torque or applies zero torque. The specialist then resumes with the correct phase clock. Across five seeds, the specialist’s advantage with respect to single+clock is reward per step without latency and remains with a 10-step hold-last latency. Thus we observe no reversal in this long-dwell setting.
| Latency | Fallback | Multi+clock reward/step | Multi shared, paired bootstrap 95% CI |
|---|---|---|---|
| 0 | none | -1.578 | |
| 1 | hold last | -1.581 | |
| 5 | hold last | -1.589 | |
| 10 | hold last | -1.598 | |
| 10 | zero torque | -1.580 |
This control simulates the effect of unavailable actions; it does not measure model-loading or inference time on hardware. A ten-step gap is only 1% of a phase, so shorter phases or slower deployment systems could yield a different ranking. If the reviewer has a particular deployment setting or latency mechanism in mind, we would welcome that example to help us delimit the appropriate model.
Response to Reviewer Mvz8 (1/2)
Q1 Under a fixed total interaction budget B, a shared policy learns from all phases, whereas each of M equally sampled specialists receives approximately B/M transitions. If that allocation leaves specialists early in their learning curves, additional interactions may benefit them more than an already-stable shared policy. At low budget, this may lead to clear underperformance of a multipolicy solution. This makes a ranking change possible for different learning budgets. For example, in an identical-phase Pendulum control (M=10, five seeds), specialist reward per step improves from −6.524 at B=5M to −0.715 at 10M and −0.429 at 20M, under the same entropy coefficient. Figures 8 and 10 represent this phenomenon with the learning curves at different budgets. In Figure 8, we added the single policy learning curve and see that multipolicy has a longer start before learning compared to a single policy. In Figure 10, at a 5M budget with 10 phases, each local policy from multipolicy underperforms compared to a single policy. This is in line with the idea of budget starvation for those local policies, which didn't have the time to learn compared to single policy.
W1 We agree that there are more number of periodic examples in the paper. We have added new experiments introducing uncertainty and aperiodicity. You can find them in the detailed results in Appendices A.4.1 and A.4.2, especially a new experiment on HAPS with structured periodicity, with the results presented in the following tables. For all of those experiments, the policy ranking didn't change compared to the periodic results. This is in line with what is presented in the paper's theory, as the structure of the aperiodicity is still known.
We added a five-seed HAP control with five recurring phases and ten occurrences of length T=500 that you can also find in Appendix A.4.1. Both architectures receive sufficient observed scheduler information. Specialist minus shared return per step is:
W2 We agree that mixture-of-experts architectures, policy specialization, and transition-aware learning have substantial prior literature. We do not claim these ideas individually as new. Our contribution is to study the choice between sharing and specialization in phase-structured RL through an explicit transient--quasi-stationary decomposition. It separates the settled specialization gain from the transient handoff cost and makes clear how phase duration and phase-local data affect the comparison.
The return decomposition is an exact accounting identity for learned policies, not a new optimization guarantee or, by itself, a practical architecture selector. Our additional contribution is to connect its terms to measurable structural diagnostics and test their predictions through controlled interventions. We would also welcome pointers to closely related work we may have missed.