- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes FIBA-Flow, an offline safe RL method that learns safety-constrained policies without complete safety-compliant trajectories. It combines TD cost estimates with short-horizon cost predictions to construct budget-constrained action distributions and is applied to a MeanFlow policy. An adaptive guide further adjusts the policy's budget conditioning through distribution matching. Intensive experiments on 38 DSRL tasks show that FIBA-Flow satisfies the average cost constraint on all tasks and achieves the highest return on 25 tasks.
Critical strengths
-
Technically sound method with reasonable design. The proposed method combines TD-based cost estimates with short-horizon cost predictions to address delayed safety costs. This design is intuitive. The budget-constrained action reweighting and adaptive budget guidance form a coherent framework for learning safe policies from unsafe trajectories. In general, the method is reasonable.
-
Extensive empirical experimental results. The paper evaluates the method on a wide range of tasks across three benchmark envs with seven baselines. It also includes comprehensive ablation studies and sensitivity analyses to investigate the effects of different components. The results and detailed analysis are convincing.
Critical weaknesses
I only raise one critical weakness about the strange motivation. The paper focuses on offline RL datasets without any complete budget-compliant trajectories. However, it is unclear how common or realistic this setting is in practice, especially since the experiments artificially construct it by removing all safe trajectories. More details about this concern are provided in Q1 below.
Questions for authors
Q1. About motivation: What practical scenarios motivate the assumption that the offline dataset contains no complete budget-compliant trajectories? This setting looks strange. In practice, offline RL datasets usually have safe demonstrations, instead of unsafe demonstrations, e.g., in autonomous driving and robot manipulation. Then when can this budget-violating dataset be useful in practice?
Q2. About the short-horizon cost critic: The short-horizon cost critics (predictors) are trained using logged subtrajectories in a supervised manner instead of TD learning. However, the deployment actions may differ from the logged actions (for example, when we compute the TD target in Q-learning, we sample a random next action from the policy instead of using the recorded next action). How does the method account for the distribution shift between these actions and the recorded behaviors?
Q3. Adaptive budget guide: The guidance mechanism actually mixes policies (actions) under different budget values, but the deployed policy uses only a single selected scale. How can we ensure that the eval scale is sufficiently covered during training? Will this discrepancy influence the cost constraint satisfaction?
I do not have many questions about the techniques and the experiments. They are clearly written and sound enough. Although there are some ambiguous expressions, such as the definition of marginal velocity in MeanFlow, adding more defensive writing is not necessary because they will damage the storytelling.
If the authors can address the above concerns, especially the motivation, I would like to raise the score to 3.
General Response
We thank the reviewers for their careful reading and constructive feedback. We have provided detailed point-by-point responses and revised the manuscript accordingly. The main updates are:
Motivation and compliant demonstrations (Reviewer pBSJ, W1/Q1 and follow-up). We expanded the motivation in Section 1 and added experiments retaining 25%, 50%, 75%, and 100% of the original budget-compliant trajectories on eight Point tasks. FIBA-Flow achieves mean normalized cost below one at every retention level. We also analyze the returns of compliant and non-compliant trajectories to help explain the observed performance differences (
Appendix A.5.3, Tables 7–8).Adaptive budget guidance (Reviewer pBSJ, Q3). We clarified how the mixture of action distributions serves as the guide’s training target and how the guide outputs a single continuous budget scale during execution (
Section 4.3).Training duration (Reviewer AHdh, Q1). We specified that FIBA-Flow is trained for 100,000 updates (
Table 3).Finite-sample MMD estimator (Reviewer AHdh, Q4). We added the estimator used to train the guide, explicitly showing the exclusion of self-pairs among the four generated actions and the weights assigned to the guide target actions (
Appendix A.3, Eq. (27)).Comparison with BCRL (Reviewer AD2J, W1/Q1). We added a comparison on all 38 tasks under the same filtering protocol and deployment budget of 10. FIBA-Flow satisfies the mean cost constraint on all 38 tasks, compared with 18 for BCRL (
Appendix A.5.1, Table 5).Short-horizon cost prediction (Reviewer AD2J, W2; Reviewer pBSJ, Q2). We clarified the supervision used to train the short-horizon cost predictor and its relationship to the cost-to-go estimate (
Section 4.1). We added an ablation replacing recorded short-horizon cost supervision with TD training. Across eight Point tasks, the two approaches achieve average NR/NC of 0.23/0.65 and 0.22/0.66, respectively (Appendix A.5.2, Table 6).Related work (Reviewer AD2J, minor comment). We expanded the discussion of GAS and CAPS and their relationship to FIBA-Flow (
Section 2).We welcome any further suggestions, including clarifications or results from our responses that the reviewers would find helpful to include in the manuscript.