- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper extends Constrained Variational Policy Optimization (CVPO) to hybrid (parameterized) action spaces. In these spaces each action is a discrete mode plus a mode-conditioned continuous control. The action space is modeled as a disjoint union of mode-specific domains, which may have different dimensions or no continuous parameter at all, with a sum–integral reference measure.
Critical strengths
- Constrained hybrid-action RL has few dedicated methods. The variational framework suits it: the nonparametric E-step reweights a joint measure over modes and controls without relaxing the discrete choice, and the projection handles mode-dependent control dimensions.
- Assumptions are stated precisely for the theories. The E-step proof handles primal attainment (via Dunford–Pettis) and the boundary rather than assuming them away.
- Strong results against the baselines. Learning is fast on all four tasks; reward reaches about 2 on Moving and HardMove within roughly 120k steps.
Critical weaknesses
- Novelty relative to existing variational hybrid-action RL is overstated. Neunert et al. (2020) are cited here only as an application example. Yet they already apply MPO’s E/M procedure to hybrid continuous–discrete policies, and split the M-step trust region into separate continuous and discrete KL constraints so the two parts can be controlled independently.
- Statistical evidence is thin relative to the claims. With 3 seeds, several cells hinge on a single seed. Parking d = 1.0 (23.26 26.94) is consistent with two seeds near 38.8 and one failing near -8. Feasibility is judged on the across-seed mean, which can hide infeasible final policies. Take Moving d = 1.0 (0.85 0.97): if all three non-negative per-seed costs satisfied the limit and averaged 0.85, their sample SD could be at most ≈0.26. So at least one of the three H-CVPO policies violates the constraint, even though the setting counts as feasible.
- Reproducibility concerns. The following settings are missing:
- how is set from and scheduled over iterations;
- the values of , , , , the particle counts , and the update frequency;
- Computational cost. The E-step enumerates every mode with particles. That is fine for HardMove's 16 modes but not for the combinatorial mode spaces in the motivating applications (e.g., unit commitment).
Reference
Neunert et al., 2020, "Continuous-discrete reinforcement learning for hybrid control in robotics"
Questions for authors
Can you reconcile your baseline results with those reported by Zhou et al. (2025)? Specifically: same task implementations and cost scaling, same budget, their released code and hyperparameters? What explains CHPO’s identical results across limits?
Reference
Zhou et al., 2025, "CHPO: Constrained hybrid-action policy optimization for reinforcement learning"