acceptodds
Withdrawn from ICLR 2027

Action Shaping: Policies Absorb What They Can Express

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

20 views

Abstract

Reward shaping has a theorem: a potential-based term can be removed without changing the optimal policy. The same practice on the action channel, an offset added in training and dropped at deployment, has no theorem. Nothing cancels an action offset, so the correction is kept at deployment or removed without a guarantee. We call it ***action shaping*** and state its principle. **A trainable policy absorbs an offset its own output layer can reproduce exactly**, which is what we mean by express; what is absorbed can be removed with the return intact. Its minimal instance is a zero-initialized linear head behind a learnable gate, added to an actor that trains through a learned action-value function, with no penalty or schedule. The gate rises and then falls on its own, for deterministic and stochastic actors alike, and on 20 tasks removing the head costs almost nothing. **The condition is exact reproduction, not capacity**: a nonlinear head with more parameters is not absorbed, and in a paired control, one linear path added to a nonlinear base head restores absorption. Exact reproduction gives the loss a flat direction that gradient noise drifts along, and the offset's amplitude indicates, before removal, what dropping the head will cost. Action shaping thus gains the counterpart of the shaping theorem, a condition for absorption, together with the mechanism behind it and a diagnostic that reads it. **Policies absorb what they can express, and only that.**

Primary area:
reinforcement learning
Keywords:
reinforcement learning, action shaping, reward shaping, actor-critic, residual policy learning, implicit regularization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_bpEgOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors introduce action shaping, a mechanism to add a learned offset to actions. This action head is progressively "absorbed" by the main policy network as it's weight is decayed over training. They tests their method on SAC, TD3 and PPO on many continuous action game but do not compare it to the baselines (without action shaping), and the experiments on Atari do not allow me to conclude if their method is worth the extra computational complexity.

Critical strengths

The paper explores action shaping, and tries to understand how to absorb this shaping into the main policy (I don't know why the policy could not stay with the learned bias).

Critical weaknesses

I have absolutely no clue why are the authors doing this in the first place. The paper lacks strong motivation, it reads a lot like "reward shaping exists, why not action shaping?". The authors do not identify a problem in which this method would help. What is the motivation? The motivation behind reward shaping is e.g. to guide the policy in situations where reward is too sparse.

They also do not compare to any baseline, not even the base method (SAC, PPO, TD3) on which they plug in action shaping. The experiments do not show any base performance impact, but only that removing the action head is not too costly. Why should it be removed? For edge devices? Also, your first question should target base performances. Before showing that removing the action-offset does not impact performances:

  1. Show that the policy with action offset has better performances than the baselines (with clear second order metrics).
  2. Motivate why the action offset should be dropped.

I don't say that there is no motivation to such a work, it's just not explicitly outlined in the paper.

On clarity: In Figure 1, place that alpha is going to decay over time. Would tremendously help understanding faster this first Figure (and the whole paper).

Questions for authors
  1. What is the motivation behind action shaping? What existing problem are you solving?
  2. Why is there no comparison to the baselines TD3, SAC and PPO in terms of reward?
Reviewer_h4jzOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks at whether an extra action head, added to the actor during RL training, can simply be thrown away at inference without any loss in performance. The method, called Action Shaping, attaches a linear head to the actor and scales its output with a learnable scalar gate.

What the authors find is that this extra head helps early on but slowly stops mattering. Its gate drifts toward zero by itself, without any regularization or schedule pushing it to zero. They call this phenomenon absorption and argue that it is enabled by the base policy's ability to exactly reproduce the auxiliary head's contribution.

The paper provides a theoretical analysis tying absorption to flat directions in the loss landscape and to gradient noise. Experiments cover TD3, SAC, and PPO on 20 continuous-control tasks, plus discrete SAC on 26 Atari games, and a series of ablations investigate the conditions under which absorption occurs.

Critical strengths
  • Interesting and well motivated problem. The paper studies an interesting optimization question: can a policy automatically absorb what a training time auxiliary component was contributing? The approach used in the paper is simple, requiring only an additional linear head and a learnable gate.
  • Thorough and convincing experiments. The method is tested across several algorithms and environments, with five random seeds. The seven-arm ablation is particularly interesting as it investigates the effects of expressibility, frozen versus trainable base policies, shared versus separate representations, and linear versus nonlinear heads, and together these results make a strong case for the proposed absorption mechanism.
  • Interesting theoretical angle. The authors tie the observed behavior to flat directions in the loss landscape and to the implicit regularization that comes from the gradient noise. This theoretical analysis offers a plausible explanation for why the the auxiliary gate might decrease without an explicit regularization term.
Critical weaknesses
  • The practical benefit of spontaneous absorption is not entirely clear. As the authors point out in Section 2.3, the linear shaping head can be merged exactly into the based head by combining their weights and this works at any point in training, whether or not the gate has reached zero. Therefore, while the spontaneous absorption phenomenon is interesting from an optimization perspective, it's hard to see what it buys you over just merging the heads at deployment.
  • The theoretical explanation relies on relatively restrictive assumptions. The analysis of gate decay relies on an idealized setting with SGD, label noise and a squared-error objective, whereas the RL experiments use Adam, non linear policies and learned critics. The experiments are consistent with the explanation, but it's not obvious how directly the theory carries over to the real RL training.
  • The writing is not super clear, some passages are a bit dense and could be streamlined to make the paper easier to follow.
Questions for authors

1 - The proposed explanation emphasizes exact expressibility as the main condition for absorption. However, all experiments use a learnable scalar gate. Would the same phenomenon happen with an ungated additive linear head, which is also exactly expressible by the base head? This could help clarify whether absorption is a general consequence of expressibility or depends on the specific parametrization.

2 - How does Action Shaping compared with unmodified TD3 and SAC in final performance and sample efficiency across the 20 continuous-control environments?

3 - Appendix E shows that a small terminal residual does not necessarily imply that the auxiliary head can safely be removed. Although the authors discuss using action-space offsets instead, could they evaluate how reliably this alternative diagnostic predicts removal costs across the full benchmark, especially when evaluated on unseen states?

Reviewer_8Gn5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates whether "action shaping" is possible: analogously to reward shaping, does there exist an offset we could implement to the action space that is as useful as reward shaping?

Critical strengths

None

Critical weaknesses

The paper is unfortunately impossible to parse and read. Jargon is introduced everywhere; many sentences are intricate and hard to read. Moreover, the question per se is not interesting, and it ignores much previous literature in RL about exploration or Bayesian RL.

Anyway, I could not understand what the authors were trying to explain, and the paper seems completely AI-generated. The mathematical parts of the paper introduced notation from nowhere, without explaining what each component means. The theoretical statements are also imprecise and very hard to parse for their actual meaning and implications.

I strongly recommend rejection.

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.