acceptodds
Withdrawn from ICLR 2027

Beyond the Policy Trust Region: Attributing and Controlling Representation-Induced Movement in PPO

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Trust-region methods constrain how far a policy moves, but they do not reveal which part of the policy network produces that movement. In policies with a learned representation and an action head, updates to the two components can reinforce or compensate for one another, so the complete old-to-new policy change can hide substantial component-level movement. We make these dynamics observable by recombining old and new representations with old and new action heads to form four counterfactual policies. In particular, evaluating the new representation through the frozen behavior head measures the action-distribution change attributable to the representation update. Building on this construction, we introduce the Movement-Attributed Representation Trust Region (\MART), a plug-in regularizer that limits excess movement along this representation path while preserving the underlying PPO-family actor objective. Across extensive experiments in open source environments from various control scenarios, \MART reduces the deviation of the attributed movement to the representation and the action ratio of the sample in all the settings tested. In traffic control, these reductions become stronger as each rollout is reused for more optimization epochs, and the action head often moves farther when the representation path is constrained, revealing compensation that is partially hidden by complete-policy diagnostics. Moreover, the results show that controlling policy movement generally benefits from considering not only the complete policy update, but also the component through which that movement arises.

Primary area:
reinforcement learning
Keywords:
Reinforcement learning, Proximal policy optimization, Policy gradient, Trust region

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_XRV3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how representation updates and action-head updates change a PPO policy. It combines old and new representations with old and new action heads to measure their effects. The authors then propose MART to limit large action-distribution changes caused by representation updates under the old action head. The experiments show smaller policy changes and better task results in some settings. Although the authors have made good efforts, some issues still need to be addressed.

Critical strengths
  1. The paper provides a clear way to examine changes in different parts of the policy network. The four-policy comparison shows how representation and action-head changes may offset each other. This helps explain changes that are not visible from the complete policy alone.

  2. The experiments cover several control tasks and PPO-family methods, using five paired seeds. The authors report both policy changes and task results. They also show cases where smaller changes do not improve performance, which helps readers understand the limits of MART.

Critical weaknesses
  1. It is not clear why the representation-induced change needs to be limited. A large change under the old head does not by itself show a problem. The representation and action head may simply adjust together while the final policy remains similar. The paper needs more evidence that this change is linked to later training problems.

  2. It is not clear whether the gains come from this specific constraint or from smaller updates in general. MART reduces several measures together, including representation-induced change, complete-policy change, and action probability ratio changes. The current results do not fully separate these effects.

Questions for authors
  1. In Sections 3.2 and 4.3, the authors should check whether larger representation-induced changes are followed by worse training results when complete-policy changes are similar. This would help explain why the additional constraint is needed.

  2. In Sections 4.5–4.7, the authors should better explain why MART helps in some settings but not others. It would be useful to compare cases where the two components offset each other without harm and cases where this is followed by a drop in performance.

  3. The authors should compare MART with a smaller learning rate for the representation network, at least in the main traffic setting. Both methods should use the same training budget. This would help show whether simply reducing the update size gives similar gains.

  4. In Section 4.6, the authors should add a penalty on the complete-policy TV or KL change. They should compare task results when the complete-policy changes are similar. This would help show what extra benefit comes from limiting the representation path.

Reviewer_RPtkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work analyses various trust-region reinforcement learning methods (PPO, SPO, DRPO) by decomposing their policy movements during training into two components: change stemming from updates to the representation space, and change stemming from updates to the action head. They then propose MART, and method to constraint representation-space-induced policy movement. They evaluate this method, in comparison to existing trust-region methods, in three RL environments.

Critical strengths
  • The method is simple and generally well explained.
  • MART appears to effectively reduce representation space movement, as it is designed to.
  • The designed metrics seem suitable.
  • Relevant baseline trust-policy methods are compared to.
Critical weaknesses
  • This work is lacking a clear motivation. It is unclear from reading it why constraining the movement of the representation space, specifically, rather than the entire policy, is necessary or desirable.
  • It is also unclear how often MART will be applicable to modern policy architectures. The authors should elaborate on how often modern policies can be broken down into representation space and action head, e.g. does this apply for flow-based policies?
  • The comparisons to PPO/SPO/DRPO seem to be for matched number of epochs. Can further analysis be given to the performance of these methods and MART across different amounts of policy movement? At the moment, it is possible that some of the reward gains of MART are simply because it generally more heavily constrains policy movement than the other methods, and running those methods for fewer epochs could have the same effect.
  • The results for Section 4.7 seem to be missing. I cannot see any reported metrics for the 'wrong-way movement' that is described.
  • Raw reward results are not presented, only deltas between the baselines are MART for each environment. Reporting the raw rewards for each method would verify that they are well trained, and achieve results similar to what has previously been achieved in existing literature.
Questions for authors
  • Can the authors elaborate on the necessity of this work? Why is it useful to constrain the movement of the representation space of a policy, specifically?
  • Can the baselines be compared to at matched policy movement, rather than matched number of training epochs? To demonstrate that reward gains cannot be explained away by this.
  • Line 464 indicates some instability for PPO-MART in training - how common was this, and how does it compare to other methods stability?
  • Can the authors should elaborate on when MART can be applied to modern policies, e.g. can it apply for flow-based policies?
  • Can the authors report the raw reward results, rather than just the deltas between methods?
Reviewer_R8bJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper study the problem of co-adoptation of representation and actor head learning in control tasks. As they interfer with each other, it is usually a good idea to stablize their updates. Practically, the paper proposes to regularize the change of action distribution induced by changing representation when action head are hold to be the old one, after examing a bunch of options. It then present some messy and likely AI generated experimental results that is assumed to be convincing.

Critical strengths
  1. Regularizing action distribution change might be useful, and a more careful study on how to regularize is a worthy research question.
Critical weaknesses
  1. The notations are badly chosen. There is no need to use o/n subscripts, please expand it out to old and new. Use z_old = f_{\theta_old}(x), z_new = f_{\theta_new}(x) . You use \bar / no \bar, and o, n to mean the same thing, while the symbol choice is not self-explanatory. Then, layed up to total variation distance with again, strange symbols. Honestly, this looks like from unpolished LLM generation. The same goes to the experiments. The presentation lacks many definition, and comparable results are usually a preferred way by AI reporting.

Please, check for all the notations, understand their meaning and make them human-readable.

  1. The method appears to be a post-hoc re-explaination of results, which can suffer from survival bias. It does not carry much insight. Especially given the questionable experiments.
Questions for authors
  1. It is unclear that eq 7 does. Is this a metric that you think can diagnose too big movement? In this case, shouldn't there be a follow up study on its' correlation with performance or what?

  2. What is r-1 ? I cannot find the definition.

  3. Are the table generated by AI? I think, AI likes to present improvements or difference not reporting original results.

  4. Why you use total variation not KL divergence, as in standard PPO regularization?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.