acceptodds
Under review as a conference paper at ICLR 2027

Hidden States as Value Gradients: The Pontryagin Structure of Recurrent Policies

Abstract

A key capability of intelligent agents is to act effectively under incomplete state observations. Recurrent policies address this by compressing observation histories into a hidden state. In this work, we show that the hidden state of a recurrent policy admits a control-theoretic interpretation: it plays the role of the co-state in Pontryagin's minimum principle, and the readout that maps it to actions implements control-Hamiltonian minimization. The hidden state thus tracks the gradient of the value function, encoding the optimality structure of the underlying control problem. We formalize this correspondence through a class of policies we refer to as co-state policies (CPs) and show that several modern recurrent cells implicitly realize this structure. The correspondence also allows for a co-state loss for actor-critic training, in which the critic's gradient serves as a target for the actor's hidden state. Empirically, we find that hidden states trained with the co-state loss encode co-state information beyond what is linearly decodable from the environment state alone, and that the loss improves policies both within and outside this class on challenging locomotion tasks, including the H1 and Berkeley humanoids. By connecting the minimum principle to recurrent memory, we provide a control-theoretic account of what hidden states compute in continuous control and a mechanism for shaping them toward optimality.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Recurrent Policies, Optimal Control, Partial Observability, Pontryagin Maximum Principle, Pontryagin Minimum Principle, Co-state Policy, State Space Models, Humanoid Locomotion

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_hyd9OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper interprets the hidden state of recurrent policies as the co-state in Pontryagin’s Minimum Principle, which along optimal trajectories equals the gradient of the optimal value function. The authors show that the update rules of many modern recurrent architectures can be written in the form of a co-state policy, and proposes an auxiliary loss to align the actor's hidden state and the critic's gradient. They show that this loss improves co-state decodability on the partially observable halfcheetah task and improves returns on humanoid locomotion.

Critical strengths
  1. This is an interesting and elegant idea to formalize hidden state of recurrent policies as the co-state in PMP.
  2. The proposed loss is simple and architecture agnostic, and is shown to consistently improve performance in humanoid locomotion task across different types of readout layer.
Critical weaknesses
  1. In Table 2, the authors only compared against linear decodability of optimal co-states from the true state, and it only reaches R² = 0.870, indicating that co-state is not a linear function of the state on this task. The hidden state could therefore achieve higher linear decodability of the co-states simply by encoding nonlinear features of the state, which doesn’t necessarily have anything to do with the co-state. Can the authors show a stronger comparison such as a nonlinear decoder of the optimal co-states?
  2. Related to point 1, the authors need more ablation experiments to convince the readers that the auxiliary loss improves episodic return by improving the encoding of co-states. Without such ablations, the networks could achieve better performance by distilling information from the critic into the actor. Can the authors provide control experiments where an auxiliary loss on the critic’s value or the critic’s hidden state (instead of the critic’s gradient) is added?
  3. For Fig 3 (especially the leftmost panel), the variability across seeds is huge and the difference across coefficient values seem to overlap with the shaded area indicating cross-seed variance. Can the authors comment on whether these differences are statistically significant, or further increase the coefficient value to bring larger effects?
Questions for authors

See above.

4 comments
AuthorsOpenReview ↗

Response to reviewer hyd9 (1/2)

We thank the reviewer for their positive feedback on the idea to formalize hidden states as co-states, and for their careful reading and comments on the presented work.

Point 1: Non-linear decoder

In Table 2, the authors only compared against linear decodability of optimal co-states from the true state, and it only reaches R² = 0.870, indicating that co-state is not a linear function of the state on this task. The hidden state could therefore achieve higher linear decodability of the co-states simply by encoding nonlinear features of the state, which doesn't necessarily have anything to do with the co-state. Can the authors show a stronger comparison such as a nonlinear decoder of the optimal co-states?

Thank you for this question about the choice of the decoder, which is a central part of our experimental design. We have implemented two non-linear baselines for the features, based on quadratic and cubic features of the state (second- and third-order polynomials).

We note, however, that a non-linear encoding of the true state cannot separate the hypothesis that the hidden states track the co-state from the hypothesis that they embed higher-order features of the state. Because the optimal co-state is itself a non-linear mapping from the state, λ⋆=∇V⋆\lambda^{\star} = \nabla V^{\star}, any non-linear features of the state run the risk of capturing the optimality conditions themselves. A linear encoding still permits a clean interpretation, namely that the hidden states encode the true state xx, which cannot be inferred from yy directly.

We compare how much a second- and third-order polynomial improves the R2R^2 score. A first-order basis would be exact if the system were linear and the cost quadratic, since a quadratic cost produces a linear co-state λ⋆=Px\lambda^{\star}=Px, where PP is the Riccati matrix. The second order captures a cubic value function, which produces a quadratic co-state, while the third order captures a fourth-order value function. Higher orders are possible, but are prone to reflecting the order required to fit the value gradient rather than testing the hidden state representations.

Our results in the table below show that the quadratic features add close to nothing to the baseline (0.8600.860 and 0.8610.861), while the cubic features raise it substantially (0.9370.937); all three are computed over the roll-outs of all recurrent cells. In the revision we report the ceiling on these same states rather than on the optimal trajectories, since that is where the decoding in Table 2 is performed; this changes the reported value from 0.8700.870 to 0.8600.860 and leaves Δceil\Delta_{\text{ceil}} unchanged. Against the cubic baseline most architectures fall below the ceiling, which is expected given that it approaches a direct approximation of ∇xV⋆\nabla_x V^{\star}.

FeaturesR2R^2 on roll-out states
Linear0.8600.860
Linear + quadratic0.8610.861
Linear + quadratic + cubic0.9370.937

Based on these results we retain the first-order baseline for Δceil\Delta_{\text{ceil}}, as it gives a directly interpretable answer to the question: how well could a direct state estimator decode co-states?

We would also like to emphasize that the h→λ⋆h \to \lambda^{\star} scores differ between policies trained with the co-state objective and those without. This difference is independent of the baseline, as the scores follow directly from the roll-outs of the trained policies.


(continues in the next comment)

AuthorsOpenReview ↗

Response to reviewer hyd9 (2/2)

(continues from the previous comment)

Point 2: Auxiliary losses

Related to point 1, the authors need more ablation experiments to convince the readers that the auxiliary loss improves episodic return by improving the encoding of co-states. Without such ablations, the networks could achieve better performance by distilling information from the critic into the actor. Can the authors provide control experiments where an auxiliary loss on the critic's value or the critic's hidden state (instead of the critic's gradient) is added?

Thank you for this suggestion. We have implemented both proposed methods: an auxiliary loss on the actor's hidden state using the hidden state of the critic, and one using the value of the critic. We evaluate these baselines on the partially observable CheetahRun task from §5.2, with the same experimental setup: a GRU cell with a linear read-out layer, an auxiliary objective coefficient of c=0.1c=0.1, and 10 random seeds per method.

The results are shown in Figure 9 of the updated draft. Both auxiliary objectives improve on the standard PPO loss, but still achieve a lower mean return than the value gradient loss. The table below reports the mean returns with their standard deviation across seeds:

Auxiliary objectiveMean returnStd (σ\sigma)
None392.1392.150.150.1
Critic value489.6489.671.971.9
Critic hidden state497.5497.565.965.9
Value gradient530.9530.941.841.8

We believe the improvement across all auxiliary losses is an interesting finding, which is in line with the idea that hidden states not only compute state estimates but also encode optimality conditions. All three objectives directly supervise the hidden state of the actor with information from the critic. That the critic's gradient is the highest-performing target is in line with our co-state policy framework, which connects the role of the hidden state to the value gradient.


Point 3: Variability across seeds

For Fig 3 (especially the leftmost panel), the variability across seeds is huge and the difference across coefficient values seem to overlap with the shaded area indicating cross-seed variance. Can the authors comment on whether these differences are statistically significant, or further increase the coefficient value to bring larger effects?

We thank the reviewer for bringing this up, and have implemented both suggested improvements: we run more seeds for each coefficient (from 10 to 15), and extend the coefficient values to include c=0.3c=0.3.

The results are shown in the newly updated Figure 4 (left panel). The table below reports the new scores, measured as the mean return over the last 10% of training per seed:

CoefficientMean returnStdSEM
0.0412.182.321.3
0.025462.253.013.7
0.05478.076.219.7
0.1535.942.110.9
0.2526.353.914.4
0.3553.147.512.3

The shaded regions in the figure show the standard deviation, that is the spread of the individual runs rather than the uncertainty of their mean, so overlap between them is expected even where the means separate. On the standard error of the mean, every non-zero coefficient separates from c=0c=0, and for c=0.3c=0.3 the standard deviation intervals no longer overlap.

The additional seeds do not change the ordering of the coefficients. The newly added c=0.3c=0.3 performs comparably to c=0.1c=0.1. We were not able to extend the sweep to larger values within the discussion period.


We hope this sufficiently addresses the questions that were raised, and we look forward to discussing these findings further.

I thank the authors for the additional experiments, which have addressed all my concerns. I'm happy to raise my score to 3.

Thank you for your careful reading and helpful suggestions. We believe they helped further improve the paper, and are thankful for your engagement throughout the discussion.

Reviewer_EuB8OpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors introduce a new control-theoretic interpretation of the hidden state in recurrent neural networks. They cast the hidden state as the co-state of the classical two point boundary value problem in control theory. This interpretation enables direct supervision of the hidden state against the gradient of the value function. The authors deploy their method as a recurrent policy for standard RL tasks and show that it improves training performance on humanoid locomotion.

Critical strengths

1.) The authors provide a new and interesting interpretation of the meaning of the hidden state of an RNN in the context of solving control tasks. I find the interpretation insightful. Beyond an interesting insight, the grounding in optimal control theory justifies an inductive bias that helps improve performance.

2.) The manuscript is presented well. Particularly Figure 1.) is a nice high level overview of the idea. Figure 2.) clearly illustrates the duality between the Hamiltonian system and the author's interpretation.

Critical weaknesses

1.) One of the key assumptions for the identification between hidden states and the costate is that the value function is differentiable. The experiments target locomotion scenarios where there are periodic contacts, so the value function is likely to be smooth. A manipulation task with irregular contact events and mode switching would help probe the limitations of this method to more general scenarios.

2.) Minor: In Figure 3 the readout plot (right) is split between two panels which compare episode return under different readouts. It effectively repeats the same information as the coefficient plot (left) especially since it does not show a significant difference between different readouts. Perhaps consider moving it to the appendix.

3.) Minor: If a reader is not already familiar with concepts of optimal control theory they might be confused by this paper. However, this may be inevitable given that the paper is grounded in theory.

Questions for authors

1.) I see that in the paper the hidden state is initialized to zero. However, if the hidden state is interpreted as the co-state then it would not necessarily be zero at the beginning. Did the authors try to initialize h as p_inv(W) @ \nabla V?

2.) Did the authors investigate removing the stop gradient on lambda in the co-state loss? Usually stop gradients prevent representation collapse, however, since lambda is already derived from the value function gradient that might prevent collapse and enable the co-state loss to assist with value training too.

3.) It would be interesting to visually observe the hidden state of the RNN with and without the co-state loss (perhaps via a TSNE plot). Does the co-state loss enable the hidden state to produce more coherent trajectories than the raw unconstrained state?

4.) The forward process of the learned dynamics is well-defined even without new observations (open-loop). Can RNNs trained with the co-state loss be run in open-loop longer than standard RNNs before the policy degrades?

5 comments
AuthorsOpenReview ↗

Response to reviewer EuB8 (1/3)

We sincerely thank the reviewer for their interest in our work, the appreciation of its novelty and for the extensive and insightful suggestions for evaluation. We address each comment individually.

Point 1: Contact discontinuity

One of the key assumptions for the identification between hidden states and the costate is that the value function is differentiable. The experiments target locomotion scenarios where there are periodic contacts, so the value function is likely to be smooth. A manipulation task with irregular contact events and mode switching would help probe the limitations of this method to more general scenarios.

Thank you for raising this. Differentiability of the value function is an important assumption that the value gradient rests on, and we agree that a manipulation task with irregular contact is a good way to test this. We have evaluated the co-state loss on the LEAP Rotate Z-axis task, where the agent has to manipulate a cube with the fingers of a LEAP hand to a desired rotation angle. Due to the task difficulty, it is required to use the asymmetric actor-critic paradigm, where the critic is given privileged state information. We adopt an LSTM policy with an MLP read-out layer, and compare different co-state coefficients c={0,0.01,0.025,0.05,0.1}c=\{0, 0.01, 0.025, 0.05, 0.1\} for five random seeds.

Our findings (Appendix E.5, Figure 11) show that small coefficients remain close to the baseline, while for the larger values c={0.05,0.1}c=\{0.05, 0.1\} the task is no longer solved. We attribute this to the co-state loss supervising the hidden states of the actor towards a target that is not defined, due to the discontinuity breaking the differentiable value function assumption. This matches the distinction the reviewer draws: on the locomotion tasks with periodic contact the loss helps, including on rough terrain, while it does not transfer to the contact-rich manipulation task with mode switching.

We note that in theory the CP framework is not fundamentally misspecified for contact-rich tasks: the co-state conditions in PMP are still well defined, but they require a sudden jump condition in their dynamics. This could be realized in the CP class by adding a network component that is permitted to make sudden jumps at moments where contact is detected. Even though contact-rich tasks fall outside of the assumed conditions, it is an important limitation that we have now placed more prominently in our work.


Point 2: Right panel of Figure 3

In Figure 3 the readout plot (right) is split between two panels which compare episode return under different readouts. It effectively repeats the same information as the coefficient plot (left) especially since it does not show a significant difference between different readouts. Perhaps consider moving it to the appendix

We agree that the right panels of Figure 4 show no clear difference between the policy read-outs. We would nonetheless argue that this is an informative finding, because the CP class makes predictions about both the hidden state and the read-out layer. The right panel is informative in the sense that it shows that the read-out layer does not tell us where the effect of the co-state loss lies, while the left panel locates it in the hidden state component. It also suggests that the co-state objective can be applied to an existing recurrent policy without having to modify its read-out layer. For this reason we prefer to keep the panels in the main text, but we are open to further discussion.


Point 3: Readability outside a control background

If a reader is not already familiar with concepts of optimal control theory they might be confused by this paper. However, this may be inevitable given that the paper is grounded in theory.

We agree with this, and would like the work to be accessible to readers from the broader machine learning community. To this end, we have added a table in Appendix B that overviews the notation, together with a brief verbal explanation. We have also extended the optimal control coverage throughout Section 3, in order to connect the PMP to concepts more familiar in the RL community: the Bellman equation and value-based methods. We hope this lowers the barrier for readers without a background in optimal control.

(Continues in the next comment)

AuthorsOpenReview ↗

Response to reviewer EuB8 (2/3)

(continues from the comment above)

Suggestion 1: Hidden state initialization

If the hidden state is interpreted as the co-state then it would not necessarily be zero at the beginning. Did the authors try to initialize h as pinv(W)p_{inv}(W) @ ∇V\nabla V?

This is a wonderful idea: hidden states in recurrent policies are indeed initialized to zeros without much theory behind it. The CP formulation suggests that they should not be zero, since the co-state trajectory is determined by a two-point boundary value problem (TPBVP) in which the initial co-state is the unknown to be solved for.

We have implemented the proposed initialization scheme, as well as two other initialization choices, and evaluated them on the partially observable CheetahRun task (§5.2), using a similar set-up (GRU policy and linear read-out layer). We first consider the reviewer's proposed pseudo-inverse of the weights multiplied by the value gradient of the initial observation, and compare it to a baseline of initializing the hidden state to zero. We also consider two further baselines: a constant initial hidden state, and an MLP Iθ(y0)I_{\theta}(y_0) that initializes the hidden state from the initial observation. The latter can be interpreted as an amortized solution to the shooting problem of the TPBVP, replacing standard Newton solvers for the initial co-state with a learned network that estimates the initialization fitting the optimality conditions of PMP.

The results are presented in Figure 8 (Appendix E.2), and show the episode return across environment steps over ten random seeds. Despite the theoretical motivation for these schemes, none of the three alternatives improves over the zero initialization. The constant initialization is marginally slower to improve across environment steps, but at convergence shows comparable performance. The two approaches that leverage the initial observation perform relatively poorly: we hypothesize that the initial observation might not contain enough information to capture the hidden state.

Despite the absence of empirical gains on the CheetahRun task, we still argue that hidden state initialization remains a promising direction that needs more consideration. The CP class provides a theoretical lens to understand why a zero or alternative initialization does or does not lead to improvements in episode return. We have incorporated this in our discussion section as follows:

A promising direction of future work is to investigate the role of hidden state initialization. While the standard practice of initializing the hidden state at zero comes with little theoretical motivation, the CP class prescribes that the initial hidden states follow from the value gradient, h(t0)=T∇xV⋆(x0)h(t_0) = T \nabla_x V^{\star}(x_0), which is not available at run time. While early experiments with learned and pseudo-inverse approximations of h(t0)h(t_0) did not yet improve performance (Appendix E.2, Figure 8), we think the theoretical implications of the CP class for hidden state initialization are worth further study.


Suggestion 2: Removing the stop-gradient

Did the authors investigate removing the stop gradient on lambda in the co-state loss? Usually stop gradients prevent representation collapse, however, since lambda is already derived from the value function gradient that might prevent collapse and enable the co-state loss to assist with value training too

This is a promising idea that could potentially improve learning. We have tested this on the partially observable CheetahRun task (§5.2), under the same conditions (GRU, linear read-out, identical training hyperparameters):

MethodEpisode return
With SG541.0±45.0541.0 \pm 45.0
Without SG441.1±119.3441.1 \pm 119.3

CheetahRun (no velocity) mean returns with and without the stop gradient (SG) operator, mean ± σ\pm\ \sigma over 10 random seeds.

The results show that removing the stop gradient operator does not result in improved returns during training. The final mean return is closer to that of the standard recurrent PPO agent, trained without the co-state loss. We have added the mean return as a function of environment steps to the appendix of the updated version.

(Continues in the comment below)

AuthorsOpenReview ↗

Response to reviewer EuB8 (3/3)

(Continues from the comment above)

Suggestion 3: Visualization of the hidden states

It would be interesting to visually observe the hidden state of the RNN with and without the co-state loss (perhaps via a TSNE plot). Does the co-state loss enable the hidden state to produce more coherent trajectories than the raw unconstrained state?

We thank the reviewer for this suggestion, and have visualized the hidden states of the RNN with and without the co-state loss on the FitzHugh–Nagumo task, using a GRU cell and dh=64d_h=64. Instead of t-SNE, which breaks the linear relationship, we directly consider the linear projection from the results in Table 2. This stays close to the theory and still allows for a clear visual interpretation of the hidden states and their behaviour. We have additionally added the optimal co-state trajectories, found using numerical optimal control, in the same figure, as well as the decoded prediction based on the methods from Table 2.

The result, presented in the newly added Figure 3, shows a clear difference between the policy trained with the co-state objective and the one without it. Without the co-state objective, the hidden states saturate at the limits of the tanh⁡\tanh and abruptly switch between the limits multiple times. The decoded co-states do not closely follow the true optimal co-states in the early and middle stages and contain sudden bumps, obtaining R2=0.86R^2=0.86. With the co-state objective, the hidden states show more diverse and smoother behaviour, and the projection follows the true optimal co-state more closely, which is reflected in the explained variance: R2=0.96R^2=0.96. These scores are consistent with the results reported in Table 2, as the test-time trajectory depicted is taken from one of the seeds.


Suggestion 4: Open-loop performance

The forward process of the learned dynamics is well-defined even without new observations (open-loop). Can RNNs trained with the co-state loss be run in open-loop longer than standard RNNs before the policy degrades?

This is a highly interesting question. To test this, we have designed an experiment that replicates the open-loop setting by freezing the last mm observations of a task, since a pure open-loop is not possible: the recurrent cell requires an observation per time step. During the last mm steps of the trial the observations are not updated, so the policy has to rely exclusively on its internal state. We use the trained policies from §5.2 (GRU with linear read-out on the partially observable CheetahRun task), and consider how the performance degrades by gradually increasing m={50,100,…,400}m = \{50, 100, \ldots, 400\}.

The findings are depicted in the newly added Figure 7, and show that the policy trained with the co-state objective loses performance more slowly. For every considered value of mm, it achieves a higher episode return during the frozen-observation sequences than its PPO counterpart. Because this score naturally decreases as mm decreases, since there are fewer time points to obtain reward from, we also consider the return within the frozen window relative to what the same policy would earn over those same mm steps at its closed-loop rate. This measure also shows that the co-state objective helps retain performance in the absence of new observations.


We hope this sufficiently addresses the questions and suggestions that were raised, and we look forward to discussing these findings further.

Reviewer_UcumOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The article details evidence for an interpretation of the hidden state participating in recurrent policies as co-states that appear in Pontryagin's minimum principle (PMP). The minimum principle describes optimal control as necessarily minimizing a certain Hamiltonian function that is applied to the optimal trajectory and co-state pairs, with the co-state evolving under affine dynamics. As the hidden state found in popular recurrent architectures, such as certain variations of the GRU and LSTM, are updated by affine operations, this motivates the question of whether these states encode the co-state described in the PMP to search for optimal actions. To this end, the authors show that under certain assumptions, a class of co-state policies (CP) can be defined that exploits the hidden state as co-state information and the action readout as Hamiltonian minimization. Empirical evidence suggests that adding a loss function that encourages a linear relationship between the hidden state and co-state improves the mean return under certain benchmarks. This shows potential utility in this interpretation in a diverse set of situations found in humanoid robotics. Surprisingly, the readouts for all readout architectures improved, including ones that lie outside of the CP class.

Overall, the article is written in a clean, readable manner. There are few issues with notation, logical clarity, figures and layout, with the appropriate proofs detailed in the appendix. The article's most prominent contribution is the proposed link between the hidden state found in recurrent architectures and an important structural quantity detailed in control through the PMP. Most interestingly, by encouraging the hidden state to encode the co-state up to some linear map, the mean return improves on all benchmarks. It would be more illuminating if the rewards considered and their potential role in producing the results in Figure 4 were discussed. Although Proposition 3.3 considers cost functions that have quadratic control cost, the experiments could utilize more sophisticated schemes. Even though the co-state is broadly estimated through the critic's gradient, more information on what role the reward structure has on the phenomena depicted in Figures 3 and 4 would be more enlightening. Furthermore, additional elaboration for why the scheme works well for the MLP readout despite it not falling into the CP class would also be helpful. Perhaps the experimental result that the co-state loss improves even within the MLP regime suggests that there is something more fundamental to the relationship between the hidden state and co-state beyond the co-state policy connection.

As a minor comment, although the authors have mentioned a possible connection with safe reinforcement learning, more explicit evidence or arguments connecting their interpretation to practical issues arising in control of cyber-physical systems would be an immediate extension. For example, evidence showing that their interpretation and co-state loss function can lead to noticeable improvements in safety outcomes for representative benchmarks would give credence to the utility of their scheme. With the previous comments in mind, overall the article would benefit from more insight into the generality and limitations of their framework or why the co-state loss function yields seemingly broad improvements. In other words, although empirical evidence shows that the PMP can be a handy and informative tool to interpret the hidden state, the potential avenues in which the community can utilize this connection in their work in a broadly applicable fashion remain unclear.

Critical strengths
  • Connections to the Pontryagin principle are novel
Critical weaknesses
  • None are critical
Questions for authors

Please address any minor concerns raised in the review.

2 comments
AuthorsOpenReview ↗

Response to reviewer Ucum (part 1/2)

We would like to thank the reviewer for the constructive points of feedback. Below, we address the concerns raised.

Point 1: Reward structure

It would be more illuminating if the rewards considered and their potential role in producing the results in Figure 4 were discussed. Although Proposition 3.3 considers cost functions that have quadratic control cost, the experiments could utilize more sophisticated schemes. Even though the co-state is broadly estimated through the critic's gradient, more information on what role the reward structure has on the phenomena depicted in Figures 3 and 4 would be more enlightening

Thank you for raising this question: making the role of the cost function more explicit is a valuable improvement. As noted by the reviewer, Proposition 3.3 only requires the control cost function c(u)c(u) to be quadratic, in order for the minimization of the control-Hamiltonian to have a closed-form solution. Experimentally, we cover the following control cost functions:

  • The FitzHugh–Nagumo (FHN) task (§5.1) uses a purely quadratic cost.
  • The CheetahRun task (§5.2) uses a state-dependent cost.
  • The H1 and Berkeley humanoid tasks (§5.3) both use a mix of exponential cost terms for the state and quadratic cost terms for the action, summed together.

Except for CheetahRun, these cost functions fit the assumption of Proposition 3.3. A read-out layer designed under a quadratic cost assumption therefore minimizes the control-Hamiltonian, u=arg⁡min⁡u∈U{c(u)+u⊤Gθ(y)h}u = \arg\min_{u \in \mathcal{U}}\{c(u) + u^{\top} G_{\theta}(y) h\}, for the FHN and humanoid tasks. This type of policy is common in standard RNN architectures, and becomes more principled under the presented framework.

CheetahRun is the only task without a quadratic control cost, as the cost function is purely state-based. Theoretically this implies a bang-bang optimal solution, where the actions switch abruptly between the actuator's limits. Under the CP formulation the control cost affects only the read-out and not the hidden state equation, so only the read-out layer needs to be adapted. Consistent with this, Figure 4 shows that the linear, selective and MLP policy heads reach comparable performance when the co-state loss is applied, suggesting that these read-out layers suffice in practice. In Figure 5 both tasks share the same cost function family, so the differences in return are caused by differences in model architecture.

Lastly, a possible way to design a policy suited to a state-dependent cost is a bang-bang policy head (inspired by Appendix Figure 6) that uses the sign of the hidden states to determine the control inputs. Such a policy could reduce the complexity of the head, but may be prone to numerical instabilities during training, since small differences in the hidden state lead to large and sudden changes in the actions. We believe this is an interesting avenue of future work that could provide a practical design principle for recurrent policies.

We have incorporated this discussion in the revised version in §5.2 and §5.3, where blue text indicates changes from the initial version.


Point 2: Why does the co-state loss work outside the CP class?

Additional elaboration for why the scheme works well for the MLP read-out despite it not falling into the CP class would also be helpful. Perhaps the experimental result that the co-state loss improves even within the MLP regime suggests that there is something more fundamental to the relationship between the hidden state and co-state beyond the co-state policy connection.

While the CP class prescribes what type of read-out layer can be enough to represent an optimal solution, it does not make claims about which architectures perform well in practice.

The co-state loss in particular applies only to the hidden state, and not directly to the read-out layer. An MLP read-out can moreover represent the linear read-out as a special case, so falling outside the CP class describes what the read-out is constrained to rather than what it is able to represent. Empirically this matches our finding that the choice of read-out matters less than the coefficient of the loss (Figure 4, right panel).

We agree with the reviewer that this points towards something more fundamental. Within the PMP framework the optimal action depends on the state only via the projection g(x)⊤λ⋆g(x)^{\top}\lambda^{\star}, which is therefore a sufficient statistic for the optimal action. While the CP class is one construction in which this statistic is linearly available, a different read-out may well capture it too. The fundamental relationship is about what the hidden states represent, rather than the exact architecture prescribed by the CP class. We believe this is why the co-state loss performs well on policies that fall outside the CP class.

We have incorporated this discussion in the updated submission, with changes marked in blue.


(Continues in the next comment)

AuthorsOpenReview ↗

Response to reviewer Ucum (part 2/2)

(Continues from the previous comment)

Point 3: Cyber-physical systems, generality and limitations

More explicit evidence or arguments connecting their interpretation to practical issues arising in control of cyber-physical systems would be an immediate extension. For example, evidence showing that their interpretation and co-state loss function can lead to noticeable improvements in safety outcomes for representative benchmarks would give credence to the utility of their scheme.

Overall the article would benefit from more insight into the generality and limitations of their framework or why the co-state loss function yields seemingly broad improvements. In other words, although empirical evidence shows that the PMP can be a handy and informative tool to interpret the hidden state, the potential avenues in which the community can utilize this connection in their work in a broadly applicable fashion remain unclear

The connection to cyber-physical systems is an important point that can improve the impact of the work. We would like to discuss two settings where we believe the work can lead to direct improvements: safe reinforcement learning, and event-based control.

Safe recurrent policies. In safety-critical settings the control-Hamiltonian can be augmented to include state constraints. The CP class provides a blueprint for designing recurrent cells that capture safety directly from this formulation:

h˙=−bθ(y)−Fθ(y,u) h−Sθ(y,u)\dot{h} = -b_{\theta}(y) - F_{\theta}(y,u)\,h - S_{\theta}(y,u)

The network Sθ(y,u)S_{\theta}(y,u) encodes the constraint term, so that the hidden state represents the co-state of the constrained problem. This mirrors the design of PMP under state constraints, where multipliers μ\mu enter the co-state equation. The structural connection between CPs and PMP therefore suggests concrete architectural modifications for safe recurrent reinforcement learning.

Event-based control. Event-based control updates the observations at irregular times, triggered by a condition rather than by a clock. Such sensors can operate at higher frequencies while demanding fewer evaluations of the policy, but pose challenges as most standard policies are applied at regular intervals. The CP class is derived in continuous time and fits this setting naturally; the S4 and S6 cells in Table 1, for example, admit arbitrary time discretizations.

Between events the policy has to rely on its own internal state. In correspondence with reviewer EuB8 we considered an open-loop experiment that tests this, where the observations are kept frozen for an increasingly long duration so that the policy relies exclusively on its hidden state (Appendix E, Figure 7). The co-state loss helps retain performance over longer periods without new observations, which suggests a benefit in the event-based setting. The event-triggered paradigm thus allows a direct extension with the co-state objective, as it relies on the alignment of the hidden states.

Generality and limitations. An important limitation concerns the smoothness of the value function, which the value gradient rests on. Many RL tasks satisfy this assumption, but contact-rich tasks are an important class that does not. Our humanoid locomotion experiments show that the co-state loss is still effective under some forms of contact discontinuity, improving performance even on rough terrain.

In response to reviewer EuB8 we considered a task where the contact discontinuity is far more pronounced: the LEAP Cube Rotate Z-axis task, where the goal is to move the fingers of a hand to rotate a cube to a desired angle. Our preliminary findings (Appendix E, Figure 11) show that small co-state coefficients remain close to the baseline, while larger ones prevent the task from being solved, which is consistent with the smoothness assumption failing. The CP framework is not fundamentally misspecified for contact-rich tasks: the co-state conditions in PMP remain well defined, but require a jump condition in their dynamics. This could be realized in the CP class by adding a network component that is permitted to jump at moments where contact is detected. Contact-rich tasks fall outside the assumed conditions, and this is an important limitation that we have now placed more prominently in the work.

We have incorporated the discussion of the smoothness assumption and the extensions to the safe and event-based settings in the updated draft, where blue text marks new additions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: summary of revisions

We sincerely thank all the reviewers for their interest in the article and positive words regarding the concept. We have responded to all comments, questions, and concerns, and have implemented several additional experiments. We have uploaded a new version of the work in which all newly added text and figures are in blue, and removed text is in red.

In particular, the following new experiments and changes have been made:

  • New experiment: contact-rich manipulation. We evaluate the co-state loss on the LEAP Rotate Z-axis task across five co-state coefficients, probing the differentiability assumption in a setting with irregular contact and mode switching (Appendix E.5, reviewer EuB8).

  • New experiment: alternative auxiliary targets. We supervise the actor's hidden state with the critic's value and with the critic's hidden state, as controls for the value gradient (Appendix E.3, reviewer hyd9).

  • New experiment: hidden state initialization. We compare the zero initialization against a constant, a learned MLP, and the pseudo-inverse scheme W+∇yVW^{+}\nabla_y V (Appendix E.2, reviewer EuB8).

  • New experiment: removing the stop-gradient. We train with and without the stop-gradient on the co-state target (Appendix E.4, reviewer EuB8).

  • New experiment: robustness under frozen observations. We freeze the observation at increasing time steps and measure the return the policy earns on its hidden state alone (Appendix E.1, reviewer EuB8).

  • Non-linear decoding baselines. We extend the decoding ceiling of Table 2 with quadratic and cubic features of the state (reviewer hyd9).

  • Additional seeds and coefficients. Figure 4 (left) is recomputed with 15 seeds per coefficient and extended with c=0.3c = 0.3 (reviewer hyd9).

  • New figure: hidden state representations. Figure 3 shows the hidden states, the optimal co-states and their decoding, with and without the co-state loss (reviewer EuB8).

  • Notation table and extended background. Appendix B lists all notation, and Section 3 connects the minimum principle to the Bellman equation and value-based methods (reviewer EuB8).

  • Extended discussion. We state the contact-rich limitation prominently together with the hybrid extension of PMP, and add the implications of the CP class for safe reinforcement learning and event-based control (reviewer Ucum).

  • Other changes. The horizontal axis of Figure 4 is corrected: the conversion from iterations to environment steps understated the training budget. The linear decoding ceiling in Section 5.1 is now measured on the states each policy visits rather than on the optimal trajectories, matching where the decoding in Table 2 is performed.

Taken together, these additions leave the conclusions of the paper unchanged. The effect of the co-state loss persists with 15 seeds per coefficient, and the control experiments show that supervising the hidden state with the critic's value or hidden state helps less than supervising it with the value gradient. The manipulation experiment shows where the method stops working: once contact discontinuities break the differentiability of the value function, the co-state loss no longer helps. We state this as a limitation in the discussion.

We thank the reviewers once again, and hope all points have been sufficiently addressed. We look forward to continuing the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.