- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Orthogonal Advantage Decomposition Policy Optimization (OADPO), a method that learns a light weight value function using independent features as control variate to reduce variance of policy optimization. The method first selects a small set of pre-specified state features, models their effects on expected return, and shrinks each component according to the estimated noise. Under independent features/noise assumptions, the decomposition yields separately optimal shrinkage weights. The learned estimator serves as a baseline in a discounted step-level advantage term added to GRPO, allowing information to be shared across states without requiring identical observations as in GIGPO.
Critical strengths
- The theoretical results provide a principled approach for fitting and regularizing the baseline under some independence assumptions, and may be particularly useful when low dimensional state features are available a priori.
- The method addresses a limitation of trajectory-level feedback without requiring exact observation grouping (e.g., GIGPO).
- The practical algorithm is computationally lightweight, and achieves competitive results compared to other baselines.
Critical weaknesses
- The theoretical results are based on assumptions (e.g., Assumption 1) that can be very difficult to verify or satisfy in practice. As such, it is unclear to what extent the algorithms actually retains the optimality guarantees.
- While feature selection is automated, the feature pool is still handcrafted. It is unclear how this could generalize to other domains where such features are hard to get (e.g., math reasoning).
- Some practical algorithm design appears unjustified. For example, the addition of two different advantage estimates (eqn 4).
Minor: The emphasis on "critic-free" is potentially misleading as the method is essentially learning a lightweight value function. Clarifying that the method avoids a separate neural critic would be more precise.
Questions for authors
- Can the authors justify the additive combination of advantages from GRPO and OADPO? Does this update correspond to a well-defined optimization objective?
- The theoretical results establish MSE optimality under stated assumptions. How does this translate into reduced policy-gradient variance or improved policy optimization?
- How does OADPO compare with a simple regression baseline using the same features and discounted targets?