acceptodds
Withdrawn from ICLR 2027

From Advantage Perturbations to Counterfactual PPO Guidance

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Proximal Policy Optimization (PPO) relies on advantage estimates to reinforce useful actions, but in sparse-reward tasks these estimates can be delayed and noisy, making it difficult to identify which earlier actions led to success. An external oracle, such as a planner, expert, safety checker, or simulator, can provide action-level information not available from the environment reward. We study advantage perturbations as an interface for adding oracle information directly to PPO’s policy update, without changing the environment reward or executed actions. We show that state-only perturbations do not shift the expected local policy gradient at the rollout policy, whereas action-relative perturbations can change preferences between actions at the same state. Controlled CartPole experiments confirm this mechanism and show that reversing the perturbation reverses the learned preference for a corrective action. We then use task-informed manual oracles to guide PPO on four sparse-reward MiniGrid tasks. Since manually specifying preferred actions requires task-specific knowledge, we next introduce a more general oracle based on local counterfactual action evaluation, which compares the outcomes of alternative actions from selected rollout states. Beyond adding this counterfactual signal at the advantage level, we also test it in the TD update and directly in the policy-loss objective. These interventions show task-dependent effects relative to standard PPO, with improvements in several task–intervention combinations.

Primary area:
reinforcement learning
Keywords:
Proximal Policy Optimization, Advantage Perturbations, Counterfactual Reasoning, Sparse-Reward Reinforcement Learning, Policy Steering

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_4KJMOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper considers the scenario where external information from an oracle is available that can be useful in policy optimization, and proposed a way to perturb the advantage function to improve learning efficiency. The authors show that only action-dependent perturbations can affect policy gradients, and empirically evaluate two ways of designing such perturbations (i.e., manual and local counterfactual). Results show that using these perturbations can speed up learning.

Critical strengths

The paper is overall clearly written, and the empirical evaluations are well presented (i.e., uncertainty quantification, detailed algorithms/hyperparameters). The cartpole example also provides useful intuition for understanding the contribution.

Critical weaknesses
  1. The theoretical result (contribution 2) that state-dependent functions do not change the expected policy gradient is well-known.
  2. The motivation is a bit unclear to me, why should we augment the advantage function rather than doing reward shaping? It appears to me that TD-CF can be viewed simply as changing the reward function, and, considering the fact that the policy gradient becomes biased in both cases, I don't see why we should prefer .
  3. The ability to "teleport" to an arbitrary state is a very strong assumption, and I'm not sure how practical this is beyond simulators. Moreover, it is not clear to me whether CF guidance offers benefits beyond providing additional advantage estimates through rollouts of alternative actions. Finally, the empirical comparison is not entirely fare considering that CF uses more rollouts.
Questions for authors
  1. When should we prefer perturbing advantages over reward shaping?
  2. Do the reported training steps include all counterfactual simulator transitions? If not, how do CF-based methods compare with PPO under matched total interaction budgets?
Reviewer_9YaROpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies external action-level guidance for PPO in sparse-reward tasks. It adds policy-centered oracle scores to the actor advantage while retaining the environment reward and critic targets. Its local analysis shows that state-only offsets cancel from the expected gradient, while action-relative scores can steer it, and gives a sufficient alignment condition for counterfactual guidance. CartPole experiments test these effects and reverse the direction of a targeted preference. On four MiniGrid tasks, it evaluates manual symbolic planners and counterfactual scores generated by restoring visited states, forcing candidate actions, and averaging bootstrapped continuations under a frozen policy. The authors compare advantage-level guidance with TD-level guidance, which also changes critic targets, and an all-action loss-level intervention. They report faster learning with manual guidance and task-dependent gains from the counterfactual variants.

Critical strengths

The intervention locations are clearly distinguished; code and logs are available.

Critical weaknesses
  1. Trivial theoretical result. Proposition 1 directly follows the standard policy-gradient baseline identity and offers little theoretical novelty.

  2. Overlap with vine sampling. CF restores visited states, forces each candidate action, and samples continuations under the frozen rollout policy, closely following TRPO's vine sampling (Section 5.2). No comparison establishes a benefit over directly using the same estimates.

  3. Limited CartPole validation. Little observed effect for the tested state-only perturbation does not establish that arbitrary state-only advantage perturbations leave learning unaffected. Such perturbations can change gradient variance and clipped PPO updates.

  4. Limited informativeness of manual-oracle gains. Section 4 claims that task-aligned guidance accelerates sparse-reward learning. The oracle encodes task dynamics and shortest-path completion costs, so improvement is expected. This demonstrates that designed guidance can help; the particular advantage-injection interface remains untested against alternatives using the same oracle.

  5. Unmatched simulation budgets. CF branches are excluded from the plotted budget. Supplied Advantage-CF logs show five-seed mean total/ordinary transition ratios of 21.04, 21.22, and 24.38 at training end for DoorKey, KeyCorridor, and UnlockPickup. Extra sampling could explain the gains; matched-budget controls are missing.

Questions for authors
  1. Does additive guidance outperform directly using the same vine-based estimates?
  2. Do gains persist against PPO with matched total simulation budgets?
  3. What do the manual-oracle gains establish beyond the expected benefit of task-specific guidance? Is there a comparison using the same oracle through another interface?
Reviewer_KWRXOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper adds an external oracle score to the advantage estimation in PPO to ineject action-level guidance while keeping the same rewards and actions. the paper shows that only the action-relative (policy-centered) part of such a perturbation shifts the expected policy gradient, since the state-only part acts as a baseline.

Critical strengths
  • The decomposition into state-only vs. action-relative parts is a clean way to think about this, and the paper is easy to follow.
  • The x sweep separates “action-dependent” from “just bigger,” and the sign-flip experiment is a nice sanity check.
Critical weaknesses

The comparison to PPO isn’t compute-matched. The CF methods run extra simulator rollouts at every queried state: every action, 2 rollouts each, up to H steps. By my count that’s roughly q·|A|·R·H, or about 20–45× extra environment steps per real step depending on the task, and none of it shows up on the x-axis. This matters because the main empirical claim is that CF guidance makes PPO learn faster. Given how modest and mixed the gains in Table 2 are, I don’t think they would survive giving PPO the same interaction budget. As it stands, I can’t tell whether the method helps or just uses a lot more data.

Questions for authors

What happens if PPO gets the same total number of env steps, counting the CF rollouts?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.