- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes an offline safe RL method (C-Steering). A bounded latent actor selects inputs to a MeanFlow decoder trained exclusively by behavior cloning. This paper calibrates the discounted cost critic to an episodic budget via an amortized per-step cost shift. Second, it uses a bounded full-dimensional latent prior for the decoder to avoid representational degeneracy associated with spherical latents. Across 21 DSRL tasks and two budgets, C-Steering satisfies the mean cost budget in 37/42 task–budget cells, including 31 with positive normalized return.
Critical strengths
-
The paper articulates and operationalizes three complementary design choices for constrained latent-space steering: budget calibration via an amortized TD cost target; a full-dimensional, bounded latent prior that removes a measure-theoretic degeneracy of spherical priors; and a direct, bounded, single-pass actor.
-
Changing only the threshold of the unshifted cost critic increases budget-satisfying cells from 7 to 38, while the tested multiplier-update variants reach 7–15. The paper also explains why shifted targets change optimization scaling as well as the threshold. This provides actionable evidence about which design choice matters within the studied framework.
Critical weaknesses
- The amortized cost target only yields equivalence under restrictive assumptions (constant per-step cost, no true terminations) and is otherwise a surrogate without guarantees; constraint satisfaction still relies on empirical validation.
- The dual update uses dataset-state values rather than on-policy states, and the paper observes multiplier saturation/oscillation; formal analysis of bias or remedies is limited to empirical observations.
- The main configuration was selected using the evaluation suite, limiting independent evidence of generalization.
- There are 38 tasks on DSRL benchmark, but the paper only report 21 tasks and 2 budgets, which is not enough.
- Baseline fairness: the main method is trained for 500k steps; some baselines (e.g., OSRL, O3SRL) were run with their official 100k steps, and not all baselines were examined at 500k, which can bias comparisons.
Questions for authors
- Do you have results for a sphere decoder retrained at the ball’s radius, or for a boundary-restricted actor using the ball-trained decoder?
- Are there existing results with configurations selected on separate development tasks, or with longer training for additional constrained baselines?
- Can you disentangle the effect of maximal radius from interior support by re-training the sphere with the same maximal radius as the ball?
- Can you report an additional OPE-style safety audit to further assess off-support risks of selected actions?