acceptodds
Under review as a conference paper at ICLR 2027

Beyond Grouping: Orthogonal Advantage Decomposition for Critic-Free LLM Agent RL

Abstract

Group Relative Policy Optimization (GRPO) assigns the same rollout-level advantage to every step, which obscures credit assignment in multi-step LLM agents. Step-level methods such as GiGPO provide finer credit only when the same observation appears in multiple rollouts, but such exact matches are often sparse: in one ALFWorld reproduction, roughly one third of step groups had no match early in training. To move beyond exact grouping, we propose Orthogonal Advantage Decomposition Policy Optimization (OADPO), a critic-free method that replaces step grouping with a feature-based decomposition of expected return. OADPO shares evidence across nonidentical states through common feature components and assigns each component a closed-form shrinkage weight based on its estimated signal-to-noise ratio, allowing informative components to be trusted while noisy ones are suppressed. Because all steps in a rollout share the same outcome reward, OADPO corrects the noise estimate at the rollout level rather than treating steps as independent samples. Under independent features, the resulting weights are optimal within this estimator family and yield strictly lower prediction error than both GRPO's group mean and per-state averaging. The fitted state-dependent control variate is combined with a discounted step-level target, providing finer credit assignment without an additional critic, rollout, or model forward pass. Across ALFWorld, WebShop, and ScienceWorld with three model backbones, OADPO achieves the highest final success rate in seven of nine settings, including 96.4% on ALFWorld with Llama-3.2-3B, while matching GRPO's per-step compute in a WebShop timing check. An ALFWorld ablation further shows that the fitted control variate contributes a 6.0-point gain beyond the discounted step target alone.

Primary area:
reinforcement learning
Keywords:
LLM agents, reinforcement learning, credit assignment, advantage estimation, critic-free policy optimization, control variate

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_axrbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Orthogonal Advantage Decomposition Policy Optimization (OADPO), a method that learns a light weight value function using independent features as control variate to reduce variance of policy optimization. The method first selects a small set of pre-specified state features, models their effects on expected return, and shrinks each component according to the estimated noise. Under independent features/noise assumptions, the decomposition yields separately optimal shrinkage weights. The learned estimator serves as a baseline in a discounted step-level advantage term added to GRPO, allowing information to be shared across states without requiring identical observations as in GIGPO.

Critical strengths
  1. The theoretical results provide a principled approach for fitting and regularizing the baseline under some independence assumptions, and may be particularly useful when low dimensional state features are available a priori.
  2. The method addresses a limitation of trajectory-level feedback without requiring exact observation grouping (e.g., GIGPO).
  3. The practical algorithm is computationally lightweight, and achieves competitive results compared to other baselines.
Critical weaknesses
  1. The theoretical results are based on assumptions (e.g., Assumption 1) that can be very difficult to verify or satisfy in practice. As such, it is unclear to what extent the algorithms actually retains the optimality guarantees.
  2. While feature selection is automated, the feature pool is still handcrafted. It is unclear how this could generalize to other domains where such features are hard to get (e.g., math reasoning).
  3. Some practical algorithm design appears unjustified. For example, the addition of two different advantage estimates (eqn 4).

Minor: The emphasis on "critic-free" is potentially misleading as the method is essentially learning a lightweight value function. Clarifying that the method avoids a separate neural critic would be more precise.

Questions for authors
  1. Can the authors justify the additive combination of advantages from GRPO and OADPO? Does this update correspond to a well-defined optimization objective?
  2. The theoretical results establish MSE optimality under stated assumptions. How does this translate into reduced policy-gradient variance or improved policy optimization?
  3. How does OADPO compare with a simple regression baseline using the same features and discounted targets?
Reviewer_iFdrOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes Orthogonal Advantage Decomposition Policy Optimization (OADPO), a critic-free method for step-level credit assignment in LLM agent reinforcement learning. Instead of grouping identical observations, it estimates a state-dependent baseline using shared state features. It decomposes this baseline into feature effects and interactions, applies signal-to-noise-based shrinkage weights, and accounts for correlated samples within trajectories. Candidate features are primarily hand-designed for each environment and automatically selected during warm-up. The resulting baseline is combined with a discounted step-level target and the standard GRPO advantage. Experiments across ALFWorld, WebShop, and ScienceWorld with three model backbones report the highest mean success rate in seven of nine settings. An ALFWorld ablation separates the contributions of the discounted target and fitted baseline.

Critical strengths
  1. The paper provides a relatively broad experimental evaluation.
  2. The appendix discusses numerous limitations in detail, and the authors have made substantial efforts to support reproducibility.
Critical weaknesses

I believe this paper has substantial methodological weaknesses and poor generalizability, and should be clearly rejected.

  1. The method relies heavily on manually constructed features, which appear to contribute substantially to the gains in the ablation experiments. This raises fairness concerns because competing methods, such as GRPO, are not provided with the same engineered features. It also limits generalizability: the features must be redesigned for each new benchmark. For example, applying the method to Sokoban would require constructing another feature set.
  2. Even with the engineered features, several reported improvements over the strongest baseline are modest, particularly on WebShop and with the 8B backbone. Although Table 1 reports means over three training seeds, it does not provide standard deviations, confidence intervals, or statistical significance tests for the main comparisons. Consequently, the evidence does not establish whether these small gains are reliable or reflect training and evaluation variability.
  3. Although the authors disclose their AI usage extensively on page 10, the manuscript appears to me to have been generated directly by AI, particularly the appendix. It contains numerous expressions that I associate with AI-generated writing and would not expect from a human author or AI closely supervised by one, including “FEATURE CATALOGUE AND THE NO-LEAKAGE CONTRACT” (line 1085), “EVERY AXIS WE VARIED” (line 1244), and “WHERE THE SIGNAL LIVES” (line 1297). The appendix also contains numerous identifiers not introduced elsewhere, such as alfworld_rtg09.json (line 1066) and webshop_match_rtg09.json (line 1076). While AI-assisted academic writing is becoming increasingly common, this does not mean that a manuscript appearing to be entirely AI-generated should be accepted.
Questions for authors

In lines 491–492, you state that AI was used “to draft sections of the paper: drafting parts of the text.” Could you clarify whether the appendix was directly generated by AI, and whether substantial portions of the main text were also AI-generated?

Reviewer_f7ajOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Critic-free methods like GRPO suffers from poor credit assignment in multi-step rollouts, by assigning the same advantage to each step in the rollout independently of their value. Previous approaches like GiGPO introduces Critic-free per-step advantages, by computing the advantage with a per-step baseline averaging the return across all the same states occurring in a group. For this to work well, multiple occurrences of the same states must happen regularly. The authors report independent analysis (i.e., Wang et al., 2026a) that this is often not the case, at least for the ALFWorld benchmark. To overcome this limitation, the authors introduce OADPO, a per-step advantage, by correcting the group mean return (i.e., GRPO baseline function) for each step based on the return for a set of independent discrete features and their pair combination. This enables to share the baseline return across nonidentical states (i.e., don't require an exact match), additionally, assigning each feature (and pair of features) a weight based on its estimated signal-to-noise ratio (i.e., down-weighting the return for noise features and up-weighting that of reliable ones), which the authors show how to estimate in a closed-form solution without any learning. In practice, OADPO subtracts the per-step baseline explained above from a discounted per-step return, while added together with the GRPO standard advantage. OADPO outperform previous baselines across 3 benchmarks and for different models.

Critical strengths
  1. Clean formulation with a nice closed-form solution to set the weights with no need for tuning. OADPO also provides a theoretical trade-off between GRPO and GiGPO, guaranteeing lower prediction error of the baseline than both extremes in ideal conditions.
  2. Good benchmark results in terms of both sample efficiency (reaches top performance fastest) and success rate, albeit with smaller margin for larger models.
  3. They include comparison to previous methods (e.g., GiGPO, HGPO) as well as useful ablation, enabling to separate the contribution of the discounting effect from the control-variate baseline in OADPO.
  4. They consider not only the effect of single feature for predicting the per-step baseline correction but also pair interactions.
  5. The motivation, method and results are well explained.
Critical weaknesses
  1. Most of the performance contribution seem to come from the per-step discounted return, not from the control variate, which is the main contribution of the paper (e.g., GiGPO also introduces a discounted return). This also affects the novelty of the paper, where its novel component does not provide large benefits.
  2. The pool of feature for each environment is hand specified, although the authors acknowledge this.
  3. Lack of some natural baseline comparison to better discriminate the contribution of each component (see questions).
  4. The added complexity of the control variate may not fully justify the performance benefits (i.e., 6 points on ALFWorld).
  5. The statistics are based on small sample size (i.e., 3 seeds for performance and 1 seed for design sensitivity). Additionally, single-seed differences below 5 points were not interpreted, when the control variate was shown to introduce a 6 point improvement on two additional seeds.
Questions for authors
  1. How does OADPO performance compare to a critic-based method (e.g., PPO) ?
  2. How did you ensure the performance benefits originating from OADPO control variate come from the proposed orthogonal feature decomposition and not simply from hard-coding domain knowledge into the advantage via the hand picked pool of features (i.e., guiding credit assignment)? Did you try any experiment where you hard code the same feature knowledge into the advantage without any orthogonal decomposition ?
  3. Could you compute the OADPO vs GiGPO performance gap across different group sizes, quantifying the state repetition rate (e.g., by keeping a counter of exact state matches per-group) ? assessing whether OADPO advantage increases under fewer repetitions.
  4. The ultimate goal of a good baseline is to reduce the variance of the update, did you try to quantify this via some proxy (e.g., the cosine similarity between gradients of two random splits of a mini-batch), compared to the b^=0\hat{b}=0 ablation on the same batches ?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.