- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper interprets the hidden state of recurrent policies as the co-state in Pontryagin’s Minimum Principle, which along optimal trajectories equals the gradient of the optimal value function. The authors show that the update rules of many modern recurrent architectures can be written in the form of a co-state policy, and proposes an auxiliary loss to align the actor's hidden state and the critic's gradient. They show that this loss improves co-state decodability on the partially observable halfcheetah task and improves returns on humanoid locomotion.
Critical strengths
- This is an interesting and elegant idea to formalize hidden state of recurrent policies as the co-state in PMP.
- The proposed loss is simple and architecture agnostic, and is shown to consistently improve performance in humanoid locomotion task across different types of readout layer.
Critical weaknesses
- In Table 2, the authors only compared against linear decodability of optimal co-states from the true state, and it only reaches R² = 0.870, indicating that co-state is not a linear function of the state on this task. The hidden state could therefore achieve higher linear decodability of the co-states simply by encoding nonlinear features of the state, which doesn’t necessarily have anything to do with the co-state. Can the authors show a stronger comparison such as a nonlinear decoder of the optimal co-states?
- Related to point 1, the authors need more ablation experiments to convince the readers that the auxiliary loss improves episodic return by improving the encoding of co-states. Without such ablations, the networks could achieve better performance by distilling information from the critic into the actor. Can the authors provide control experiments where an auxiliary loss on the critic’s value or the critic’s hidden state (instead of the critic’s gradient) is added?
- For Fig 3 (especially the leftmost panel), the variability across seeds is huge and the difference across coefficient values seem to overlap with the shaded area indicating cross-seed variance. Can the authors comment on whether these differences are statistically significant, or further increase the coefficient value to bring larger effects?
Questions for authors
See above.
4 comments
Response to reviewer hyd9 (2/2)
(continues from the previous comment)
Point 2: Auxiliary losses
Related to point 1, the authors need more ablation experiments to convince the readers that the auxiliary loss improves episodic return by improving the encoding of co-states. Without such ablations, the networks could achieve better performance by distilling information from the critic into the actor. Can the authors provide control experiments where an auxiliary loss on the critic's value or the critic's hidden state (instead of the critic's gradient) is added?
Thank you for this suggestion. We have implemented both proposed methods: an auxiliary loss on the actor's hidden state using the hidden state of the critic, and one using the value of the critic. We evaluate these baselines on the partially observable CheetahRun task from §5.2, with the same experimental setup: a GRU cell with a linear read-out layer, an auxiliary objective coefficient of , and 10 random seeds per method.
The results are shown in Figure 9 of the updated draft. Both auxiliary objectives improve on the standard PPO loss, but still achieve a lower mean return than the value gradient loss. The table below reports the mean returns with their standard deviation across seeds:
| Auxiliary objective | Mean return | Std () |
|---|---|---|
| None | ||
| Critic value | ||
| Critic hidden state | ||
| Value gradient |
We believe the improvement across all auxiliary losses is an interesting finding, which is in line with the idea that hidden states not only compute state estimates but also encode optimality conditions. All three objectives directly supervise the hidden state of the actor with information from the critic. That the critic's gradient is the highest-performing target is in line with our co-state policy framework, which connects the role of the hidden state to the value gradient.
Point 3: Variability across seeds
For Fig 3 (especially the leftmost panel), the variability across seeds is huge and the difference across coefficient values seem to overlap with the shaded area indicating cross-seed variance. Can the authors comment on whether these differences are statistically significant, or further increase the coefficient value to bring larger effects?
We thank the reviewer for bringing this up, and have implemented both suggested improvements: we run more seeds for each coefficient (from 10 to 15), and extend the coefficient values to include .
The results are shown in the newly updated Figure 4 (left panel). The table below reports the new scores, measured as the mean return over the last 10% of training per seed:
| Coefficient | Mean return | Std | SEM |
|---|---|---|---|
| 0.0 | 412.1 | 82.3 | 21.3 |
| 0.025 | 462.2 | 53.0 | 13.7 |
| 0.05 | 478.0 | 76.2 | 19.7 |
| 0.1 | 535.9 | 42.1 | 10.9 |
| 0.2 | 526.3 | 53.9 | 14.4 |
| 0.3 | 553.1 | 47.5 | 12.3 |
The shaded regions in the figure show the standard deviation, that is the spread of the individual runs rather than the uncertainty of their mean, so overlap between them is expected even where the means separate. On the standard error of the mean, every non-zero coefficient separates from , and for the standard deviation intervals no longer overlap.
The additional seeds do not change the ordering of the coefficients. The newly added performs comparably to . We were not able to extend the sweep to larger values within the discussion period.
We hope this sufficiently addresses the questions that were raised, and we look forward to discussing these findings further.
I thank the authors for the additional experiments, which have addressed all my concerns. I'm happy to raise my score to 3.
Thank you for your careful reading and helpful suggestions. We believe they helped further improve the paper, and are thankful for your engagement throughout the discussion.
Response to reviewer hyd9 (1/2)
We thank the reviewer for their positive feedback on the idea to formalize hidden states as co-states, and for their careful reading and comments on the presented work.
Point 1: Non-linear decoder
Thank you for this question about the choice of the decoder, which is a central part of our experimental design. We have implemented two non-linear baselines for the features, based on quadratic and cubic features of the state (second- and third-order polynomials).
We note, however, that a non-linear encoding of the true state cannot separate the hypothesis that the hidden states track the co-state from the hypothesis that they embed higher-order features of the state. Because the optimal co-state is itself a non-linear mapping from the state, λ⋆=∇V⋆, any non-linear features of the state run the risk of capturing the optimality conditions themselves. A linear encoding still permits a clean interpretation, namely that the hidden states encode the true state x, which cannot be inferred from y directly.
We compare how much a second- and third-order polynomial improves the R2 score. A first-order basis would be exact if the system were linear and the cost quadratic, since a quadratic cost produces a linear co-state λ⋆=Px, where P is the Riccati matrix. The second order captures a cubic value function, which produces a quadratic co-state, while the third order captures a fourth-order value function. Higher orders are possible, but are prone to reflecting the order required to fit the value gradient rather than testing the hidden state representations.
Our results in the table below show that the quadratic features add close to nothing to the baseline (0.860 and 0.861), while the cubic features raise it substantially (0.937); all three are computed over the roll-outs of all recurrent cells. In the revision we report the ceiling on these same states rather than on the optimal trajectories, since that is where the decoding in Table 2 is performed; this changes the reported value from 0.870 to 0.860 and leaves Δceil unchanged. Against the cubic baseline most architectures fall below the ceiling, which is expected given that it approaches a direct approximation of ∇xV⋆.
Based on these results we retain the first-order baseline for Δceil, as it gives a directly interpretable answer to the question: how well could a direct state estimator decode co-states?
We would also like to emphasize that the h→λ⋆ scores differ between policies trained with the co-state objective and those without. This difference is independent of the baseline, as the scores follow directly from the roll-outs of the trained policies.
(continues in the next comment)