acceptodds
Under review as a conference paper at ICLR 2027

UniQ: Unified Conditional Flows for Offline Safe Reinforcement Learning

Abstract

Offline safe reinforcement learning seeks to maximize reward under cost constraints using a fixed dataset. Lagrangian methods repeatedly adjust a multiplier, changing the policy objective and potentially destabilizing learning when cost estimates are inaccurate. We propose , a conditional flow framework that replaces continued multiplier adaptation with a fixed safety rule estimated from offline data. The rule prioritizes reward in safe regions, balances reward and cost near the safety boundary, and prioritizes cost reduction in unsafe regions. Conditioned on state and accumulated cost, a single flow critic learns the combined return and guides a conditional flow policy, with both models conditioned on the physical state and accumulated cost. To train these models across a wider range of budget conditions, we resample cumulative costs for each logged transition. Our analysis establishes population Bellman consistency and fixed-policy convergence under vanishing update error, and identifies the utility-change term eliminated by the fixed safety rule. A capacity-restricted construction further demonstrates a strict value-error advantage over separately fitted component critics. On nine MetaDrive tasks, satisfies the cost threshold on every task and achieves approximately higher average normalized reward than the strongest baseline. [Anonymous Project Page & Videos](https://anonymous4699-uniq-project.static.hf.space/)

Primary area:
reinforcement learning
Keywords:
Flow Matching, Reinforcement Learning, Safe Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_29pCOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

UniQ learns an offline policy that maximizes reward subject to a cost limit. It first assigns each state to a safe, boundary, or recovery region and freezes the reward–cost weights. One flow critic then fits the combined return, and one conditional flow policy learns from that value. Both are conditioned on accumulated cost, which is resampled during training. On nine MetaDrive tasks the reported normalized cost is below 1 on every task. The mean normalized reward is 0.59, against 0.40 for FISOR, the strongest baseline that is feasible on all nine tasks.

Critical strengths

The contribution is to learn both the combined critic and the policy as conditional flows. The critic fits the reward–cost return directly, and the policy is trained from that value under a frozen safety rule, conditioned on accumulated cost. On all nine MetaDrive tasks the normalized cost stays below 1, and the mean normalized reward is 0.59, about 48% above FISOR. The costs lie between 0.90 and 0.97, so the higher reward is obtained while still meeting the constraint.

Critical weaknesses

The objective depends on a boundary weight and on a second check that uses the cost already spent. Neither formula is given, so the return the flow is fitting cannot be reconstructed. The higher reward under the cost limit is also not tied to one piece of the method: the experiments never remove the flow critic, the frozen rule, or the resampling of accumulated cost. The method is complicated, and it depends on many hyperparameters.

Questions for authors

What is the formula for the boundary weight, and what is the check that uses the cost already spent?

If the flow critic is replaced by separate reward and cost critics, or the safety rule is allowed to keep adapting, or cost resampling is turned off, which change removes the gain?

Reviewer_6ncUOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes UNIQ, an offline safe reinforcement learning framework that replaces ongoing multiplier updates with a fixed, data-derived safety rule. A unified flow critic learns the combined reward–cost value and guides an advantage-weighted conditional flow policy. Both models condition on state and accumulated cost, with cost resampling expanding training coverage. The analysis establishes Bellman consistency, fixed-policy convergence under vanishing update error, and an approximation advantage in a capacity-restricted construction. On nine MetaDrive tasks, UNIQ achieves reported mean costs below the threshold and approximately 48% higher average normalized reward than FISOR.

Critical strengths
  1. Learning the critic and actor under the same reward–cost utility is a natural design choice. Conditioning on both the physical state and accumulated cost also allows the policy to make different decisions at the same physical state under different cost histories, which is consistent with the need to account for cost history in safe RL.
  2. Section 4.2 distinguishes utility changes, policy changes, and update errors in the value-tracking analysis. The appendices also provide concrete constructions comparing direct fitting with fitting components separately and then combining them under restricted settings. I think these analyses help explain why the authors adopt a fixed rule and directly learn the unified objective.
Critical weaknesses
  1. The paper is mainly motivated by target drift caused by changes in λ\lambda during training, and the potential errors from learning components separately and then combining them. Section 4.2 and the appendices provide theoretical analysis, but I did not find convincing experiments showing that these effects are major bottlenecks in the evaluated tasks. Figure 1 does not define the component flows or explain how the results were obtained. I can therefore only interpret it as an illustration of the authors' argument, rather than direct evidence of what happens during actual training. In particular, changing λ\lambda changes the reward–cost trade-off and the policy target, but this does not by itself establish harmful training instability. I would like to see measurements and visualizations of target drift, value-tracking error, or related training behavior to understand whether these effects substantially influence the final performance or training stability.
  2. Table 1 includes several offline safe RL baselines, and PD-Flow is explicitly described as using an adaptive Lagrange multiplier. However, the paper does not sufficiently explain its settings for the critic representation, cumulative-cost conditioning, and resampling procedure, or clarify which of these settings are shared with UniQ. I therefore cannot determine whether the performance gains come from fixing the safety rule, using a unified value representation, resampling cumulative costs, or other differences between the methods. A more clearly specified and controlled comparison would help connect the reported gains to the proposed motivation.
  3. The reported ablations only examine sensitivity to the number of cumulative-cost samples and the sampling ratio, and both analyses are limited to the three Easy tasks. These experiments retain the complete UniQ method and only show how its performance changes under different hyperparameter settings. I did not find component ablations for the fixed safety rule, unified value representation, or cumulative-cost resampling. This makes it difficult to understand the actual contribution of each design to the reported performance.
Questions for authors
  1. How exactly was Figure 1 obtained? Is it based on an analytical construction or on results from actual model training? I would like the authors to provide the definitions of the component flows, the exact role of λ\lambda, and the procedure used to generate the results, and explain how the figure relates to the actual algorithm and the analysis in Appendix G.

  2. During actual training on MetaDrive, do changes in λ\lambda cause noticeable target drift, value-tracking error, or training instability? I would like to see corresponding measurements or training curves to understand the magnitude of these effects and whether they are related to the final performance differences. This is a key step for me to be convinced by the paper's core motivation.

  3. Which settings are shared between PD-Flow and UniQ, and which are different? In particular, please clarify the critic representation, cumulative-cost conditioning, resampling procedure, model capacity, and hyperparameter tuning settings. I would like these details to be made explicit so that I can judge whether this comparison tests the benefit of freezing the safety rule. Could the authors also include additional, similarly targeted baselines?

  4. Please provide component-level ablation experiments. I consider these a basic requirement for the paper. I would particularly like to see a comparison between directly learning the unified value and separately learning the corresponding weighted components before combining them, as well as a comparison between resampling and training only on logged cumulative costs. These experiments would help me understand the actual role of each design in the evaluated tasks.

  5. What exactly is the budget-dependent check that maps κ0(s)\kappa_0(s) to κ(s,u)\kappa(s,u)? How are λω(z)\lambda_\omega(z) and εreg\varepsilon_{\mathrm{reg}} determined in the experiments, and are the same settings used across different tasks? I would like the authors to provide the complete criteria, actual parameter settings, and selection procedure, rather than only stating that these quantities are frozen before the final training stage.

Reviewer_181MOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes UniQ for offline safe reinforcement learning. The method first estimates and freezes a safety rule that assigns context-dependent reward and cost weights. A conditional flow critic then models a scalar Bellman target for the resulting combined utility, and an advantage-weighted conditional flow policy is trained from logged actions. Both models incorporate accumulated cost, and training resamples cumulative-cost inputs while maintaining consistent successor costs.

Critical strengths
  • The study is interesting, and the question is important.
  • The experiment results look good; UniQ reports the highest feasible reward on all nine evaluated MetaDrive tasks.
Critical weaknesses
  • The pretrained cost value favors low-cost behavior; identifying a state from which low-cost behavior is possible does not itself certify the safety of the subsequently learned reward-seeking policy. As the paper acknowledges, Bellman consistency and removal of the utility-change term do not establish feasibility of Equation (2). Appendix E.7 assumes that an ideal reference policy already has a sufficient safety margin. Consequently, specification and calibration of the frozen rule are essential to evaluating the practical safety claim.
  • The ablations vary the number and mixture ratio of cumulative-cost samples, but do not compare against using logged cumulative costs alone, removing cost conditioning, or replacing the unified critic with separate critics trained for the same context-weighted utility.
Questions for authors
  • Can controlled comparisons establish the contributions of direct unified critic fitting and cumulative-cost resampling?
  • Could we discuss the influence of the evaluation episode counts, cost normalization and discounting convention, baseline tuning protocol, and checkpoint-selection procedure?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.