acceptodds
Under review as a conference paper at ICLR 2027

Constrained Variational Policy Optimization for Hybrid Action Spaces

Abstract

Constrained reinforcement learning in hybrid action spaces must coordinate discrete modes and mode-conditioned continuous controls under long-horizon cost constraints. Existing constrained hybrid-action methods typically optimize factorized policies directly in parameter space, whereas Constrained Variational Policy Optimization (CVPO) separates nonparametric improvement from parametric projection in its original continuous-action formulation. We introduce Hybrid Constrained Variational Policy Optimization (H-CVPO), extending this variational framework to hybrid action spaces. Under boundedness and strict feasibility, its constrained E-step has zero duality gap and, when the dual optimum is attained at positive temperature, yields a closed-form joint target distribution over hybrid actions through exponential reward–cost reweighting. Using the exact hybrid KL decomposition, we formulate a factorized M-step that fits both policy components to this target under separate discrete and conditional-continuous trust regions. For exact E/M updates, we prove fixed-reference surrogate improvement and conditional reward–cost bounds with a sufficient feasibility-margin condition. For approximate updates, we establish the existence of a dual minimizer under continuity on a compact domain and use residual-based admissibility to determine whether to project the target. Ablations identify cost-aware reweighting as an important contributor to constraint control. Across four tasks and three cost limits, H-CVPO achieves favorable reward–cost trade-offs compared with hybrid-action baselines.

Primary area:
reinforcement learning
Keywords:
constrained reinforcement learning, hybrid action spaces, constrained variational policy optimization, hybrid-action reinforcement learning, constrained Markov decision processes

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8bXAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper extends Constrained Variational Policy Optimization (CVPO) to hybrid (parameterized) action spaces. In these spaces each action is a discrete mode plus a mode-conditioned continuous control. The action space is modeled as a disjoint union of mode-specific domains, which may have different dimensions or no continuous parameter at all, with a sum–integral reference measure.

Critical strengths
  • Constrained hybrid-action RL has few dedicated methods. The variational framework suits it: the nonparametric E-step reweights a joint measure over modes and controls without relaxing the discrete choice, and the projection handles mode-dependent control dimensions.
  • Assumptions are stated precisely for the theories. The E-step proof handles primal attainment (via Dunford–Pettis) and the η→0\eta\rightarrow 0 boundary rather than assuming them away.
  • Strong results against the baselines. Learning is fast on all four tasks; reward reaches about 2 on Moving and HardMove within roughly 120k steps.
Critical weaknesses
  • Novelty relative to existing variational hybrid-action RL is overstated. Neunert et al. (2020) are cited here only as an application example. Yet they already apply MPO’s E/M procedure to hybrid continuous–discrete policies, and split the M-step trust region into separate continuous and discrete KL constraints so the two parts can be controlled independently.
  • Statistical evidence is thin relative to the claims. With 3 seeds, several cells hinge on a single seed. Parking d = 1.0 (23.26 ±\pm 26.94) is consistent with two seeds near 38.8 and one failing near -8. Feasibility is judged on the across-seed mean, which can hide infeasible final policies. Take Moving d = 1.0 (0.85 ±\pm 0.97): if all three non-negative per-seed costs satisfied the limit and averaged 0.85, their sample SD could be at most ≈0.26. So at least one of the three H-CVPO policies violates the constraint, even though the setting counts as feasible.
  • Reproducibility concerns. The following settings are missing:
    • how εE,i\varepsilon_{E,i} is set from dd and scheduled over iterations;
    • the values of ρ\rho, τc\tau_c, τKL\tau_{\text{KL}}, ηmin⁡/ηmax⁡/λmax⁡\eta_{\min} / \eta_{\max} / \lambda_{\max}, the particle counts Kb,mK_{b,m}, MMM_M and the update frequency;
  • Computational cost. The E-step enumerates every mode with KK particles. That is fine for HardMove's 16 modes but not for the combinatorial mode spaces in the motivating applications (e.g., unit commitment).

Reference

Neunert et al., 2020, "Continuous-discrete reinforcement learning for hybrid control in robotics"

Questions for authors

Can you reconcile your baseline results with those reported by Zhou et al. (2025)? Specifically: same task implementations and cost scaling, same budget, their released code and hyperparameters? What explains CHPO’s identical results across limits?

Reference

Zhou et al., 2025, "CHPO: Constrained hybrid-action policy optimization for reinforcement learning"

Reviewer_BwAzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper extends CVPO to hybrid action spaces, where each action is a discrete mode plus a continuous parameter. The E-step reweights the current policy by reward and cost to build a target distribution. The M-step fits a factorized discrete/continuous policy to this target with separate KL limits for each part. The authors prove strong duality for the E-step, along with surrogate improvement and CPO-style reward/cost bounds. Experiments on four small hybrid-action tasks with 3 seeds show higher reward than CHPO, HPPO-Lag, and PDQN-Rco, and the cost limit is met in 10 of 12 settings. In the ablations, only removing the cost term from the E-step clearly hurts performance.

Critical strengths

Clear, careful formulation. The extension of CVPO to hybrid action spaces is written rigorously, and the assumptions are stated honestly.

Transparent evaluation protocol. Results are taken at a fixed endpoint with no checkpoint cherry-picking, and the paper reports negative findings, including CHPO’s identical numbers and its own cost violations. This gives the results some credibility, though the small number of seeds limits how much they show.

Critical weaknesses

Limited novelty. Theorem 1 is CVPO’s E-step result with the action space replaced by the hybrid space, and the proof uses no hybrid-specific structure. A factorized hybrid M-step with separate discrete and continuous KL constraints already appears in hybrid MPO (Neunert et al., 2020). The paper cites that work only as an application example and neither discusses it nor compares against it. The remaining theory is standard: the KL chain rule, EM monotonicity, CPO’s bound, and the extreme value theorem (A.4). The method therefore looks like CVPO combined with hybrid MPO, and this is the main reason for my recommendation. Ablations don’t support the claimed components. Replacing the separate KLs with a joint KL meets the cost limit in more settings (11 vs. 10) and is often better on Parking. Removing the admissibility filter gives identical results in 9 of 12 settings. The only component with a clear effect is the cost-aware E-step, which comes directly from CVPO.

Weak statistics. There are only 3 seeds, and the standard deviations are often as large as the means (e.g., Parking reward 23.26 ± 26.94, HardMove cost 2.17 ± 3.32). The feasibility counts (10 of 12) are not reliable at this sample size.

Baseline concerns. CHPO’s results are identical across all cost limits on Parking, and on Moving at d = 1.5 and 2.0. This suggests the limit may not have been applied to CHPO. The paper doesn’t compare against hybrid MPO with a Lagrangian, or against CVPO on a relaxed action space. H-CVPO is off-policy while CHPO and HPPO-Lag are on-policy, so its sample-efficiency advantage at 320k steps may come from the algorithm class rather than the proposed method.

Small-scale tasks. All four benchmarks are toy environments, which doesn’t match the industrial applications used to motivate the paper.

Questions for authors

see weakness

Reviewer_HTXHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors proposes H-CVPO, an extension of CVPO to RL problems with hybrid action spaces, where each action combines a discrete mode with a mode-dependent continuous control under cumulative cost constraints. The method alternates between two steps. A constrained E-step constructs a joint target distribution over hybrid actions by reweighting the current policy according to reward and cost estimates. An M-step then fits a factorized policy (mode selection followed by mode-conditioned continuous control) to this target, using separate KL trust regions for the discrete and continuous components. A target-admissibility filter skips the projection when the approximate target exceeds prescribed cost or KL-divergence tolerances. On the theoretical side, the authors establish strong duality for the constrained E-step under regularity assumptions and derive a closed-form target when the dual optimum is attained at positive temperature. They also prove a surrogate improvement result for exact updates and derive conditional reward–cost performance bounds adapted from CPO, including a sufficient feasibility-margin condition. Experiments on four hybrid-action benchmark tasks under three cost limits show that H-CVPO achieves the highest mean reward across all 12 settings while satisfying the cost limit in 10 of them, outperforming the baselines (CHPO, HPPO-Lag, and PDQN-Rco) in terms of the reported reward–cost trade-offs along the lines of their statistics chosen. Ablation studies indicate that cost-aware reweighting is an important contributor to constraint control, while the effects of separate trust regions and target-admissibility filtering vary across tasks and cost limits.

Critical strengths

Strengths

i) Relevance: Constrained RL with hybrid action spaces is practically important and relatively underexplored. ii) Good E-step analysis:The strong-duality proof is carefully developed as far as I can tell , including a measure-theoretic treatment of hybrid action spaces and explicit assumptions for the closed-form solution. iii) The ablation results demonstrate that incorporating cost estimates into the E-step is important for constraint control across most tested settings.

Critical weaknesses

Concerns about novelty / degree of innovation: The method is essentially CVPO applied to hybrid actions. Hybrid MPO (Neunert et al., 2020) already exists and is neither compared against nor discussed, and the theory carries over from prior work with little change. Also theory disconnected from experiments.

Questionable empirical evidence. Three seeds, large variances, and questionable baselines: CHPO’s results don’t change with the cost limit, and most baselines earn negative rewards. Statistical analysis lacking. By the authors own ablations, the separate trust regions and the admissibility filter perform no better than the simpler alternatives.

Questions for authors
  • How does H-CVPO differ from combining CVPO with the hybrid MPO of Neunert et al. (2020)? Could you include that combination as a baseline?

  • Why are CHPO’s results identical across cost limits on Parking, and on Moving at d = 1.5 and 2.0? If CHPO is mostly in cost-minimization mode, why does its Parking cost stay around 150 for 960k steps? How were the baselines tuned, and was the tuning effort comparable to H-CVPO’s?

  • Statistical significance. With three seeds and large variances, are the differences significant? Can you report more seeds, or confidence intervals or statistical tests?

  • Your ablation shows the joint KL performs as well or better. What evidence supports separate discrete and continuous trust regions as a contribution?

  • How often does the filter actually trigger during training? Removing it changes results in only 3 of 12 settings.

  • How is the E-step cost threshold set from the episodic cost limit d, and is it fixed or adapted over training? Please also report the missing hyperparameters: dual bounds, particle counts, dead-zone fraction and tolerances.

  • The implemented algorithm uses separate limits on the continuous mean and standard deviation, a λ = 0 shortcut and a prior-support safeguard. Which of the theoretical results still apply to it?

  • On realism: Given the motivating examples (hybrid vehicles, power systems), can you evaluate on at least one realistic task?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.