acceptodds
Under review as a conference paper at ICLR 2027

C-Steering: Constrained Noise-Space Steering of Behaviour-Cloned Flow Policies for Offline Safe RL

Abstract

Offline safe reinforcement learning must find high-return behaviour within an episodic cost budget from logged data alone. We propose C-Steering, a bounded noise actor over a one-step flow decoder trained only by behaviour cloning. The actor selects actions from the learned decoder's image without a competing cloning penalty. Controlled experiments examine three design choices: cost-budget calibration, decoder-prior geometry and noise-actor parameterisation. Calibrating discounted cost values to the episodic budget improves budget satisfaction substantially more than the tested multiplier-update changes. A bounded full-dimensional prior raises mean return by over a covariance-matched sphere while retaining of budget-satisfying cells, with gains in both return and seed-level safe return supported by seed and task resampling. Additional diagnostics reveal near-boundary noise selection and an approximately larger RMS spread of the decoded boundary image on two-dimensional tasks, characterising how the learned action geometry differs between priors. In a single-task comparison, a direct actor achieves higher return at every seed and lower mean cost than a reflected-flow actor, with two network passes per action instead of eleven. Across DSRL tasks and two budgets, C-Steering meets the budget on cells, with positive return, compared with and , respectively, for the baseline with the most budget-satisfying cells. These findings establish an effective constrained-steering design and provide empirical insight into its prior geometry and noise selection.

Primary area:
reinforcement learning
Keywords:
offline safe reinforcement learning, safe reinforcement learning, offline reinforcement learning, constrained Markov decision process, flow matching, generative policies, noise-space steering, Lagrangian methods

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_d7rqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an offline safe RL method (C-Steering). A bounded latent actor selects inputs to a MeanFlow decoder trained exclusively by behavior cloning. This paper calibrates the discounted cost critic to an episodic budget via an amortized per-step cost shift. Second, it uses a bounded full-dimensional latent prior for the decoder to avoid representational degeneracy associated with spherical latents. Across 21 DSRL tasks and two budgets, C-Steering satisfies the mean cost budget in 37/42 task–budget cells, including 31 with positive normalized return.

Critical strengths
  1. The paper articulates and operationalizes three complementary design choices for constrained latent-space steering: budget calibration via an amortized TD cost target; a full-dimensional, bounded latent prior that removes a measure-theoretic degeneracy of spherical priors; and a direct, bounded, single-pass actor.

  2. Changing only the threshold of the unshifted cost critic increases budget-satisfying cells from 7 to 38, while the tested multiplier-update variants reach 7–15. The paper also explains why shifted targets change optimization scaling as well as the threshold. This provides actionable evidence about which design choice matters within the studied framework.

Critical weaknesses
  1. The amortized cost target only yields equivalence under restrictive assumptions (constant per-step cost, no true terminations) and is otherwise a surrogate without guarantees; constraint satisfaction still relies on empirical validation.
  2. The dual update uses dataset-state values rather than on-policy states, and the paper observes multiplier saturation/oscillation; formal analysis of bias or remedies is limited to empirical observations.
  3. The main configuration was selected using the evaluation suite, limiting independent evidence of generalization.
  4. There are 38 tasks on DSRL benchmark, but the paper only report 21 tasks and 2 budgets, which is not enough.
  5. Baseline fairness: the main method is trained for 500k steps; some baselines (e.g., OSRL, O3SRL) were run with their official 100k steps, and not all baselines were examined at 500k, which can bias comparisons.
Questions for authors
  1. Do you have results for a sphere decoder retrained at the ball’s radius, or for a boundary-restricted actor using the ball-trained decoder?
  2. Are there existing results with configurations selected on separate development tasks, or with longer training for additional constrained baselines?
  3. Can you disentangle the effect of maximal radius from interior support by re-training the sphere with the same maximal radius as the ball?
  4. Can you report an additional OPE-style safety audit to further assess off-support risks of selected actions?
Reviewer_e7WKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes C-Steering, an offline safe RL method that learns a bounded noise-space actor to select actions through a one-step MeanFlow behavior cloning decoder. It combines a full-dimensional bounded noise prior with a budget-relative cost critic and Lagrangian updates to balance reward and safety. Empirical results show that C-Steering satisfies the budget in most tasks with a high return.

Critical strengths

The empirical performance of the proposed method is good.

Critical weaknesses
  1. The novelty of the proposed method is limited. Latent space optimization through a one-step MeanFlow decoder is proposed in LPS [1]. Bounded, full-dimensional noise support is used by ReFORM [2]. The budget shift follows [3]. The combination is reasonable, but the contribution mainly lies in empirical comparison and insights.

  2. The writing of the paper needs a lot of improvement. It is very difficult to read for human readers given its current status. For example, it lacks a clear progression from motivation to method and supporting evidence, and a lot of sentences appear abruptly without connection with other parts. Moreover, the "Support and noise generation" part, which should be an element of the proposed method, only discusses the noise generation of [2], instead of their method. The paper claims "a theoretical and empirical study of bounded noise priors and actor parameterisations", but it provides limited insight into why the proposed prior and actor parameterization should improve constrained policy optimization.

[1] Latent Policy Steering through One-Step Flow Policies.

[2] ReFORM: Reflected Flows for On-support Offline RL via Noise Manipulation.

[3] Online Optimization for Offline Safe Reinforcement Learning.

Questions for authors
  1. Could the authors clarify which algorithmic elements are new relative to LPS, ReFORM, and O3SRL, and distinguish these from the contribution of the empirical study?

  2. Could the authors proofread the paper carefully and make the flow smoother for human readers?

  3. The ball–sphere comparison changes support dimension, maximum radius, and radial training distribution simultaneously. Can the authors provide a same-radius comparison and an experiment restricting the ball-trained decoder’s actor to the boundary?

Reviewer_wjKHOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper looks at noise-space steering in offline RL with a "safety penalty" at train time.

Critical strengths

None

Critical weaknesses

This paper appears to be largely AI-written.

Even directly in the abstract: "A bounded fulldimensional prior raises mean return by 12% over a covariance-matched sphere while retaining 37 of 42 budget-satisfying cells, with gains in both return and seed level safe return supported by seed and task resampling."

This type of writing, including having not provided proper context, is pervasive throughout the paper.

There is also no direct comparisons to DSRL or ReFORM (except for the latter on one small environment).

Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.