acceptodds
Under review as a conference paper at ICLR 2027

Single or Multiple Policies for Phase-Structured Reinforcement Learning?

Abstract

Many reinforcement-learning (RL) problems are non-stationary yet structured and can be decomposed into phases, each with its own transition probabilities and reward functions. When the phase sequence is known, the common solution augments the state with information to satisfy the Markovian property and applies standard RL techniques. However, prior work finds that the multi-policy approach for different phases can outperform a single state-augmented policy shared among the phases, for reasons that remain unclear. In this work, we first show that the shared policy can theoretically achieve performance of any multi-policy solution. However, whether a multi-policy solution can perform better than the corresponding single shared policy in practice depends on function approximation, learning and optimization processes, as well as, for multi-policy solutions, the sample efficiency and loss of continuity from one policy to another. We propose a regime-based phase decomposition method to identify which policy can provide better performance. The method is based on consideration of the duration of transient system dynamics relative to the duration of the quasi-stationary period. Numerical experiments are conducted with different non-stationary RL problems to validate our four major hypotheses: (a) longer phase durations favor multi-policies, (b) the heterogeneity between phases increases the burden on single policy, (c) multi-policies need sufficient data for each phase, and (d) environment-specific transition dynamics between phases can affect which policy is preferable.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Phase-Structured Markov Decision Processes, Non-Stationary Environments, Proximal Policy Optimization, Multi-Policy

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Mvz8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies when single-policy or multi-policy approaches are more suitable for non-stationary reinforcement learning. The authors divide the environment into several phases and assume that each phase is stationary. Based on this setting, they define transient and quasi-stationary regimes and introduce quantities such as handoff cost and quasi-stationary gain to analyze the trade-offs between the two approaches. The main focus of the paper is not simply whether multi-policy performs better, but under what conditions and why one approach is preferred over the other. The authors support their analysis with experiments in three environments where phase information is observable.

Critical strengths
  1. The paper provides a useful quantitative framework for comparing single-policy and multi-policy approaches in non-stationary RL. By distinguishing between transient and quasi-stationary regimes and defining quantities such as handoff cost, the authors help explain under what conditions each approach may be preferable, rather than simply comparing their performance.

  2. The paper also provides intuitive explanations for how the analytical quantities are estimated and how they relate to the experimental results. For example, in Section 5.1 (HAPS), the authors explain how they estimate the transient length and use their assumptions to predict a crossover around T = 200. This makes the analysis easier to follow and helps connect the theoretical formulation to the observed results.

Critical weaknesses
  1. The paper states in the Introduction that the framework covers both periodic and fixed non-periodic phase structures. However, all experiments are conducted in periodic environments. Since non-periodic settings are also included in the scope of the paper, I think additional validation is needed to support the generality of the analysis.

  2. Several aspects discussed in this paper, such as policy specialization and handoff costs, have already been studied in previous work across different application domains, including robotic manipulation. Although the paper brings these findings together into a quantitative framework, it is unclear whether this provides a substantial new contribution beyond combining existing observations.

  3. The comparisons between single-policy, multi-head, and multi-policy approaches raise concerns about differences in parameter counts. Previous work, such as Chen et al. (2025), has already examined the trade-off between parameter count and performance in multi-policy RL. Although this paper includes capacity and depth ablations, it is still unclear whether the performance differences come from the policy architecture itself or from differences in model capacity.

  4. The analysis considers the behavioral cost of policy handoff but does not account for policy activation latency, which is the time needed for the next policy to produce its first action after switching. If this additional cost is considered, single-policy approaches might be preferable to multi-policy approaches in some settings, especially when switching occurs frequently. Since the paper focuses on when each architecture is preferable, I think this cost should also be considered in the analysis.

Questions for authors
  1. The paper mentions that the relative ranking of policy architectures may change as the sample budget increases. Could the authors provide a more intuitive explanation of why this might happen, or support this claim with additional experimental results? This would help clarify how the sample budget affects the relative performance of different architectures.
2 comments
AuthorsOpenReview ↗

Response to Reviewer Mvz8 (1/2)

Q1 Under a fixed total interaction budget BB, a shared policy learns from all phases, whereas each of MM equally sampled specialists receives approximately B/MB/M transitions. If that allocation leaves specialists early in their learning curves, additional interactions may benefit them more than an already-stable shared policy. At low budget, this may lead to clear underperformance of a multipolicy solution. This makes a ranking change possible for different learning budgets. For example, in an identical-phase Pendulum control (M=10M=10, five seeds), specialist reward per step improves from −6.524-6.524 at B=5B=5M to −0.715-0.715 at 1010M and −0.429-0.429 at 2020M, under the same entropy coefficient. Figures 8 and 10 represent this phenomenon with the learning curves at different budgets. In Figure 8, we added the single policy learning curve and see that multipolicy has a longer start before learning compared to a single policy. In Figure 10, at a 5M budget with 10 phases, each local policy from multipolicy underperforms compared to a single policy. This is in line with the idea of budget starvation for those local policies, which didn't have the time to learn compared to single policy.

W1 We agree that there are more number of periodic examples in the paper. We have added new experiments introducing uncertainty and aperiodicity. You can find them in the detailed results in Appendices A.4.1 and A.4.2, especially a new experiment on HAPS with structured periodicity, with the results presented in the following tables. For all of those experiments, the policy ranking didn't change compared to the periodic results. This is in line with what is presented in the paper's theory, as the structure of the aperiodicity is still known.

We added a five-seed HAP control with five recurring phases and ten occurrences of length T=500T=500 that you can also find in Appendix A.4.1. Both architectures receive sufficient observed scheduler information. Specialist minus shared return per step is:

Phase orderSharedMultipolicyPaired difference, 95% Student-tt CI
Periodic32.6641.69+9.02  [+4.55,+13.49]+9.02\;[+4.55,+13.49]
Non-cyclic32.0442.30+10.26  [+5.99,+14.52]+10.26\;[+5.99,+14.52]

W2 We agree that mixture-of-experts architectures, policy specialization, and transition-aware learning have substantial prior literature. We do not claim these ideas individually as new. Our contribution is to study the choice between sharing and specialization in phase-structured RL through an explicit transient--quasi-stationary decomposition. It separates the settled specialization gain from the transient handoff cost and makes clear how phase duration and phase-local data affect the comparison.

The return decomposition is an exact accounting identity for learned policies, not a new optimization guarantee or, by itself, a practical architecture selector. Our additional contribution is to connect its terms to measurable structural diagnostics and test their predictions through controlled interventions. We would also welcome pointers to closely related work we may have missed.

AuthorsOpenReview ↗

Response to Reviewer Mvz8 (2/2)

W3 In our work, we have mostly compared policies under equal learning budget, and we agree that this may not be enough for a fair policy comparison, since model capacity is another factor that plays a role in architecture selection. However, the two factors, i.e., model capacity and learning budget, are interdependent. Therefore, it is difficult to study both of them together; one can study one factor keeping the other fixed.

For example, under a fixed learning budget, one can test if increasing the model capacity improves the performance. Also, for a given model capacity, one can check if a higher learning budget improves the performance. Given a particular learning budget, one can go for an architecture that achieves similar performance with lower model capacity.

In our work, we have increased the model capacity of the single shared policy by increasing depth and width to see if the increased model capacity improves the performance. Our observation is that increasing the shared network’s width or depth raises its active parameter count well above that of one specialist but still under the total of all the specialists, yet none of the tested variants closes the gap in the returns between shared and specialized policies. This argues against active-network size alone explaining the result. Even increasing the depth of the single shared policy above 6 seems to reduce the average reward per step for a single policy. However, this does not establish a capacity-independent architectural effect: the specialist ensemble still has substantially greater total stored capacity, and changing depth also changes optimization difficulty.

To clarify our point, we highlight some of the results. For the default HAP M=20M=20 actor--critic models, the results from Appendix A.4.1, Table 7, give us:

ArchitectureHidden layersStored / active parametersReward/step
Single+clock2x2562 x 2560.154M / 0.154M27.06
Single+clock, wider2x5122 x 5120.571M / 0.571M24.27
Single+clock, deeper4x2564 x 2560.418M / 0.418M29.51
Single+clock, deeper6x2566 x 2560.681M / 0.681M29.04
Single+clock, deeper8x2568 x 2560.944M / 0.944M23.30
Multiheadshared 2x2562 x 256 + heads0.246M / 0.153M32.76
Multipolicy20 separate 2x2562 x 2562.862M / 0.143M42.34

W4 Thank you for identifying a cost that our handoff analysis did not include. To test the policy activation latency between different phases of the multipolicy case, we added an experiment with Phase Pendulum (M=10,T=1000M=10,T=1000) and the results of Multi policy+clock along with their advantage with respect to the Single policy+clock are presented in the table below (also in Appendix A.4.2 Table 15). Note that Single+clock acts immediately with the correct phase information. For Multi policy+clock, after each boundary, the new specialist supplies no action for dd steps; during that gap, the actor either holds the previous torque or applies zero torque. The specialist then resumes with the correct phase clock. Across five seeds, the specialist’s advantage with respect to single+clock is 0.1080.108 reward per step without latency (d=0)(d=0) and remains 0.0880.088 with a 10-step hold-last latency. Thus we observe no reversal in this long-dwell setting.

Latency ddFallbackMulti+clock reward/stepMulti −- shared, paired bootstrap 95% CI
0none-1.578+0.108  [+0.032,+0.249]+0.108\;[+0.032,+0.249]
1hold last-1.581+0.105  [+0.029,+0.246]+0.105\;[+0.029,+0.246]
5hold last-1.589+0.097  [+0.021,+0.238]+0.097\;[+0.021,+0.238]
10hold last-1.598+0.088  [+0.013,+0.229]+0.088\;[+0.013,+0.229]
10zero torque-1.580+0.107  [+0.031,+0.247]+0.107\;[+0.031,+0.247]

This control simulates the effect of unavailable actions; it does not measure model-loading or inference time on hardware. A ten-step gap is only 1% of a T=1000T=1000 phase, so shorter phases or slower deployment systems could yield a different ranking. If the reviewer has a particular deployment setting or latency mechanism in mind, we would welcome that example to help us delimit the appropriate model.

Reviewer_QVDTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether to use a single policy across all phases or a specialized policy for each phase in reinforcement learning. The environment's transitions or rewards change according to a known schedule, and the current phase is observed. The authors divide each phase into a period when the state at the start of the phase has a strong effect and a period when this effect has become sufficiently small. They describe the performance gap between shared and specialized policies in terms of specialization gains and losses from state handoff. In experiments, this paper uses Proximal Policy Optimization (PPO) to compare a shared policy, a multihead policy that shares all layers except the phase-specific output layers, and independent specialized policies. The experiments cover position control of high-altitude platforms, a pendulum with changing target angles, and a grid world with changing goals and walls. The authors report that phase duration, differences between phases, training budget, and state handoff at phase boundaries can change which policy performs better.

Critical strengths

In the grid world, this paper keeps the trained policies fixed and compares carrying the state over at phase boundaries with resetting it to reachable states. The authors show that the ranking of shared and specialized policies reverses between these conditions. This comparison is useful because it separates the effect of state handoff on the performance gap from changes caused by retraining.

Critical weaknesses
  • W1 Unclear whether the problem is with specialized policies or their training method: The shared policy is trained on rewards across all phases, but each specialized policy is trained only on rewards from its own phase. It is therefore unclear whether the handoff failures of specialized policies come from using separate policies or from excluding rewards in subsequent phases from the training objective. The appendix also tests handoff-aware specialized policies. However, it is unclear whether these comparisons include rewards or values from subsequent phases in the training objective. This paper needs a control with separate policies trained to account for rewards in subsequent phases. Otherwise, it needs to limit its conclusions to comparisons with policies trained only on rewards from their own phase.

  • W2 Limited scope of the contribution: All experiments use fixed phase orders and durations, and the current phase is correctly identified. This paper does not show whether its conclusions hold when phase transition times or phase identification are uncertain. This limits the usefulness of its findings for policy selection in such environments and the scope of its contribution.

Questions for authors
  • Q1: Could you consider adding specialized policies trained to account for rewards in subsequent phases to the comparison? This comparison would help distinguish how performance is affected by parameter sharing and by differences in training objectives.

  • Q2: My understanding is that uncertainty in phase transition times or phase identification may change whether shared or specialized policies perform better. Could you evaluate the policies with such uncertainty to check whether this understanding is correct?

2 comments
AuthorsOpenReview ↗

Response to Reviewer QVDT (1/2)

Q1 We agree that comparing shared policies against phase-local specialists conflates parameter sharing with the training horizon. To isolate this effect, we evaluated a multipolicy baseline with independent phase-specific actor-critics trained on full trajectories, allowing advantages to propagate across phase boundaries without sharing parameters. GridWorld: Accounting for future-phase rewards recovers the handoff coordination of a shared policy. The results are presented in the next table and can be found in Appendix A.4.3, Table 22. Across five matched seeds, it improves over phase-local specialists by +0.0221+0.0221, +0.0238+0.0238, and +0.0257+0.0257 reward/step for ρ∈{0.3,0.6,0.9}\rho \in \{0.3, 0.6, 0.9\}, increasing the safe-goal selection rate by +0.84+0.84 at ρ=0.9\rho = 0.9 and nearly matching the shared policy's return.

Safe reward ρ\rhoLocal safe choiceFull-return safe choiceFull −- local reward/step, paired bootstrap 95%95\% confidence interval (CI)
0.30.020.00+0.02208  [+0.01828,+0.02450]+0.02208\;[+0.01828,+0.02450]
0.60.060.30+0.02375  [+0.02223,+0.02527]+0.02375\;[+0.02223,+0.02527]
0.90.100.94+0.02570  [+0.02400,+0.02762]+0.02570\;[+0.02400,+0.02762]

HAPS: In the more complex HAPS environment, cross-phase multipolicy training underperforms both standard multipolicy and shared baselines.

Dwell time TTMultipolicyMultipolicy+clockHandoff-awareHandoff-aware+warm start
20037.4738.5031.7527.14
50046.3546.7536.6927.29

As shown in the table above (Appendix A.4.1, Table 9 of the main manuscript), for M=20M=20 with 3 seeds, Handoff-aware and Handoff-aware+warm start underperform multipolicy solutions by at least 15%15\%. Because all specialists update concurrently, each policy faces a non-stationary target as downstream policies evolve. We have added these results and a discussion of this cross-phase non-stationarity trade-off to the revised paper.

The reviewer's question and the experiments highlight an important distinction on reward design. In GridWorld, including subsequent-phase rewards largely removes the specialists' handoff failure, whereas the HAP entry-state intervention alone does not improve performance and even deteriorates it. Together, these results motivate better cross-phase credit assignment and continuation-value objectives for multipolicy training. Developing a method that coordinates specialists without sacrificing their phase-local gains is a promising direction.

Q2 We thank the reviewer for this suggestion. We think this is highly relevant, and such a study will help strengthen the results. Our main experiments assume that the current phase is observed. We added limited controls to examine departures from that assumption. In pendulum, we evaluated trained policies with an incorrect phase label persisting for an entire phase occurrence. The results can be found in Appendix A.4.2, Tables 14 and 16.

Wrong-label ppSingle+clockMulti+clockMulti −- Single, paired bootstrap 95%95\% CI
00−1.686-1.686−1.578-1.578+0.108  [+0.033,+0.249]+0.108\;[+0.033,+0.249]
0.10.1−1.800-1.800−1.702-1.702+0.098  [+0.030,+0.225]+0.098\;[+0.030,+0.225]
0.30.3−2.042-2.042−1.962-1.962+0.079  [+0.024,+0.180]+0.079\;[+0.024,+0.180]
qqjjSingle+clockMulti+clockFull-return Multi+clock
1.00-2.198-2.186-2.248
1.0100-2.172-2.149-2.158
0.80-2.104-2.069-2.186
0.8100-2.178-2.087-2.317

In the above table, as the mislabel probability rises from 00 to 0.30.3, both methods deteriorate, but the specialist-minus-shared gap remains positive (0.1080.108 to 0.0790.079 reward per step). A separate test delaying the reported phase and clock by 50 steps likewise does not reverse their ranking. Uncertain phase information does not seem to reverse policy ranking.

We thank the reviewers for this question, as it helped us to strengthen our result section H4 inside the main paper "We also train independent phase-specific actor--critics on complete two-phase trajectories, allowing PPO returns and advantages to cross the phase boundary. At ρ=0.9\rho=0.9, their safe-goal choice rate rises from 0.100.10 under phase-local training to 0.940.94, close to Single+clock's 0.960.96 (Appendix Table 22). Thus, the training objective explains much of the specialists' apparent myopia in this construction. However, this benefit is specific to the two-goal construction. Giving specialists more cross-phase training information does not universally improve performance: our HAP handoff-aware variant performs worse than standard multipolicy (Appendix Table 8). Thus, addressing handoffs requires an effective credit-assignment and training design for multipolicy."

AuthorsOpenReview ↗

Response to Reviewer QVDT (2/2)

W1 Addressed with the answers to question Q1.

W2 Addressed with the answers to question Q2. We also ran an experiment in aperiodic settings for the HAPS environment (see Appendix A.4.1). The aperiodic scenario with sufficient information didn't reverse any policy ranking.

We added a five-seed HAP control with five recurring phases and ten occurrences of length T=500T=500 that you can also find in Appendix A.4.1. Both architectures receive sufficient observed scheduler information. Specialist minus shared return per step is:

Phase orderSharedMultipolicyPaired difference, 95% Student-tt CI
Periodic32.6641.69+9.02  [+4.55,+13.49]+9.02\;[+4.55,+13.49]
Non-cyclic32.0442.30+10.26  [+5.99,+14.52]+10.26\;[+5.99,+14.52]
Reviewer_aKt1OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper asks whether a reinforcement learning agent that faces a repeating sequence of environment phases should use one shared policy or a separate policy for each phase. The authors show that a shared policy can represent any set of phase specific policies. They split each phase into an adjustment period and a settled period arguing that separate policies win when the settled gain exceeds the boundary cost. Experiments with PPO on three environments support four different hypotheses (phase length, phase diversity, data budget and boundary effects).

Critical strengths
  • The question is clear and practically relevant. Many applied problems have periodic structure.
  • The paper tests its hypotheses with controlled changes in phase length, phase number, heterogeneity and budget. It also contains an experiment (GridWorld handoff) that seems to me as a clean demonstration.
  • The paper is clear where evidence is only consistent with a hypothesis without giving a proof.
Critical weaknesses
  • The key quantities G and K are defined using the learned policies, so they cannot be computed before choosing an architecture. The proposed diagnostic is therefore only possible in hindsight.
  • The bounds in equation (8) require bounds on G and K. The paper does not explain how to obtain them without training both architectures.
  • The empirical support is narrow. Only PPO is used, and the environments are small or synthetic. There is no standard continuous control benchmark beyond a pendulum.
  • Several main figures report single runs or few seeds without confidence intervals.
Questions for authors

The shared policy receives clock information in all comparisons, while the plain multipolicy does not. The paper notes this. Can you add a fairer comparison?

1 comment
AuthorsOpenReview ↗

Response to Reviewer aKt1

Q1 We thank the reviewer for raising this point. To remove this asymmetry, we trained clock-aware specialists (multipolicy + clock) for the HAPS and pendulum experiments. Each specialist receives the same phase-age signal as the shared policy, and all variants use matched seeds and interaction budgets. The results are now reported in Appendixes A.4.1 and A.4.2 (Table 5 and Table 13) of the revised manuscript and are reproduced below.

The five-seed table mean raw reward per step for HAP at fixed M=20M=20:

TTSingle+clockMultihead+clockMultipolicyMultipolicy+clock
10031.75432.05720.87521.284
20031.26733.15131.89532.544
50027.49535.22042.35642.785
200019.61031.70145.75845.946

The five-seed Pendulum phase-count control at T=1000T=1000 :

MMSingle+clockMultihead+clockMultipolicyMultipolicy+clock
2-1.888-1.891-1.752-1.773
5-1.633-1.613-1.514-1.533
10-1.688-1.692-1.553-1.578
20-2.502-2.563-2.260-1.987

In both environments, multipolicy + clock and plain multipolicy perform comparably: the differences are not statistically significant. With this clock information in multipolicy, the ranking between multipolicy-type solutions and single-policy solutions does not change. So the advantage of multipolicy + clock therefore cannot be attributed to its access to clock information.

The original submission already included a multipolicy + clock baseline for Gridworld, where the clock did improve on plain multipolicy. We attribute this to the clock acting as an extra input that diversifies the specialists' actions and helps them escape specific failure modes, e.g., repeated loops near walls. Taken together, these results show that clock information can help a multipolicy in specific settings such as Gridworld, but it does not provide a general advantage.

W1-W2 We acknowledge the reviewer's concern. This work is a preliminary study to understand the type of architecture appropriate for phase-structured problems: a single/shared policy, a specialized multiple policy, or a shared feature extractor. The measured GG and KK terms explain performance retrospectively; they are not quantities we can reliably compute before training. We agree that an immediate next step is to develop a method with pilots to identify in advance which architecture is more relevant. However, for the HAPS problem, the example provides a vision of how to evaluate in advance the architecture decision choice without computing G and K. We are working on proposing a new method allowing policy choice without explicitly computing G and K.

W3 We agree that the evaluation is limited to PPO and does not include a standard continuous-control benchmark beyond Pendulum. The three environments were chosen to isolate different aspects of phase structure: HAP combines changing spatial demand with persistent users and platform positions; PhasePendulum changes the reward target while retaining continuous dynamics; and PhaseGridWorld lets us control boundary-state reachability and handoff cost through its topology. Across them, we vary phase duration, phase count, heterogeneity, budget, and boundary rules. We will identify broader continuous-control evaluation as future work.

W4 Based on the reviewer's suggestion, we ran additional experiments to increase the number of seeds and include confidence intervals for our results. The details can be found in the main paper result function and the appendixes A.4.1, A.4.2, and A.4.3. Extended Empirical Results and Diagnostics. The updated tables are the following: 5-8, 10-17, and 21-23.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comment to Reviewers

Dear Reviewers, We thank you all for your careful reading of our paper and your valuable feedback. We have responded to each of your comments in detail and revised the manuscript accordingly. As the discussion period is coming to an end, we would greatly appreciate it if you could let us know whether our responses have addressed your concerns. We are happy to answer any further questions. Many thanks for your time and effort in reviewing our work. Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.