acceptodds
Under review as a conference paper at ICLR 2027

Beyond Safe Demonstrations: Failure-Informed Flow Policies for Offline Safe Reinforcement Learning

Abstract

Can an offline learner satisfy a deployment cost budget without complete trajectories whose cumulative cost is within that budget? Bootstrapped cost estimates may fail to capture delayed costs that only become visible in recorded subtrajectories. We introduce FIBA-Flow, a failure-informed, budget-adaptive approach to offline safe reinforcement learning. For each candidate action, we combine a cost-to-go estimate with a short-horizon cumulative-cost prediction to obtain a cost score. We optimize target action probabilities to maximize entropy-regularized predicted return under a remaining-budget constraint on the mean cost score. A budget-conditioned flow is trained to match these target distributions for one-step action generation. A guide learns to adjust the flow's internal budget input through action-distribution matching. Its targets are constructed across budget conditions under a constraint on the predicted cost set by the actual remaining budget, using critics that account for subsequent guide decisions. Evaluations on 38 DSRL tasks show that FIBA-Flow satisfies the cost constraint on every task and achieves the highest normalized return on 25 tasks among budget-compliant methods. Our anonymous link is: https://anonymous.4open.science/r/paper-6d0be256/

Primary area:
reinforcement learning
Keywords:
Offline safe reinforcement learning, flow-matching

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_pBSJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FIBA-Flow, an offline safe RL method that learns safety-constrained policies without complete safety-compliant trajectories. It combines TD cost estimates with short-horizon cost predictions to construct budget-constrained action distributions and is applied to a MeanFlow policy. An adaptive guide further adjusts the policy's budget conditioning through distribution matching. Intensive experiments on 38 DSRL tasks show that FIBA-Flow satisfies the average cost constraint on all tasks and achieves the highest return on 25 tasks.

Critical strengths
  1. Technically sound method with reasonable design. The proposed method combines TD-based cost estimates with short-horizon cost predictions to address delayed safety costs. This design is intuitive. The budget-constrained action reweighting and adaptive budget guidance form a coherent framework for learning safe policies from unsafe trajectories. In general, the method is reasonable.

  2. Extensive empirical experimental results. The paper evaluates the method on a wide range of tasks across three benchmark envs with seven baselines. It also includes comprehensive ablation studies and sensitivity analyses to investigate the effects of different components. The results and detailed analysis are convincing.

Critical weaknesses

I only raise one critical weakness about the strange motivation. The paper focuses on offline RL datasets without any complete budget-compliant trajectories. However, it is unclear how common or realistic this setting is in practice, especially since the experiments artificially construct it by removing all safe trajectories. More details about this concern are provided in Q1 below.

Questions for authors

Q1. About motivation: What practical scenarios motivate the assumption that the offline dataset contains no complete budget-compliant trajectories? This setting looks strange. In practice, offline RL datasets usually have safe demonstrations, instead of unsafe demonstrations, e.g., in autonomous driving and robot manipulation. Then when can this budget-violating dataset be useful in practice?

Q2. About the short-horizon cost critic: The short-horizon cost critics (predictors) are trained using logged subtrajectories in a supervised manner instead of TD learning. However, the deployment actions may differ from the logged actions (for example, when we compute the TD target in Q-learning, we sample a random next action from the policy instead of using the recorded next action). How does the method account for the distribution shift between these actions and the recorded behaviors?

Q3. Adaptive budget guide: The guidance mechanism actually mixes policies (actions) under different budget values, but the deployed policy uses only a single selected scale. How can we ensure that the eval scale is sufficiently covered during training? Will this discrepancy influence the cost constraint satisfaction?

I do not have many questions about the techniques and the experiments. They are clearly written and sound enough. Although there are some ambiguous expressions, such as the definition of marginal velocity in MeanFlow, adding more defensive writing is not necessary because they will damage the storytelling.

If the authors can address the above concerns, especially the motivation, I would like to raise the score to 3.

Reviewer_AHdhOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates offline safe reinforcement learning when no complete training trajectory satisfies deployment budget constraints. FIBA-Flow combines policy-dependent cost estimates with supervised short-horizon log costs to solve the entropy-regularized action reweighting problem under budget constraints, and trains a single-step MeanFlow policy based on the resulting objective. Another guide selects the policy’s internal budget conditions by matching action distributions and employs a critic capable of accounting for subsequent guide decisions. This paper reports the average budget compliance rate across all 38 DSRL tasks and achieves the best return among compliant methods on 25 tasks after excluding complete trajectories with costs not exceeding 10.

Critical strengths
  1. The problem is practically relevant, and the method effectively utilizes local information from failure trajectories.

  2. The experiment is comprehensive, with a wide range of benchmarks covered, and it has achieved outstanding results, particularly on MetaDrive.

Critical weaknesses
  1. The process for model selection and final evaluation is not fully explained. The manuscript refers to selected checkpoints and task-specific training steps, but does not state the selection criterion, the number of evaluation iterations used for selection, whether the final evaluation was independent, or whether the baseline models received comparable tuning and selection budgets. Table 4 clearly allows budget-specific checkpoints and constrains only the three-budget mean cost; however, it remains unclear whether this rule applies equally to all baselines. This uncertainty substantially limits the interpretation of the empirical advantage.

  2. The main table reports only aggregated means from three seeds and twenty evaluation episodes. Several costs are close to the threshold, including 0.97 and 0.96 on PointButton1/2 and 0.98 on EasySparse. Please report per-seed mean costs and uncertainty estimates using evaluations independent of checkpoint selection, especially for tasks near the budget threshold.

Questions for authors
  1. How are the reported checkpoints selected, and what information and evaluation budget are used?
  2. Are final results obtained on independent evaluation seeds? What are the per-seed costs on the near-threshold tasks?
  3. Does “noise-free inference” preserve Gaussian flow sampling and remove only additional exploration noise?
  4. Which finite-sample MMD estimator is used, particularly for the policy–policy kernel term with four generated samples?
Reviewer_AD2JOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FIBA-Flow, an offline safe reinforcement learning method for learning from datasets that contain no complete trajectories under the deployment cost budget. The method combines cost-to-go estimates with predictions of recorded short-horizon cumulative costs to construct cost scores for candidate actions. Together with predicted returns, these scores are used to optimize target action distributions under a remaining-budget constraint, which supervise a budget-conditioned MeanFlow policy. A separate budget guide learns to adjust the policy's internal budget input through MMD-based action-distribution matching. The components are updated within a shared offline training loop.

In comprehensive experiments, FIBA-Flow achieves mean normalized cost below one on every task and the highest normalized return among the evaluated budget-compliant methods.

Critical strengths
  1. This paper targets a meaningful problem in offline safe reinforcement learning, where the datasets contain no complete trajectories satisfying the deployment cost budget. This setting matches practical applications where budget-compliant demonstrations are difficult or too costly to obtain.

  2. The comprehensive experiments demonstrate strong support for the effectiveness of the method. The reported balance between return and budget compliance across the evaluated tasks is strong, and the method makes good use of the available budget.

Critical weaknesses
  1. Some closely related methods are missing from the empirical comparison, and the rationale for their exclusion is unclear. In particular, C2IQL and BCRL are cited briefly in the introduction but not included in the experimental comparison. It would help if the baseline selection were explained.

  2. The method section could explain the motivation behind key design choices more clearly. Some designs are specified mathematically, but their rationale and intended effects are less clearly explained. For example, more intuition for the cost-score construction in Eq. (4) would improve readability.

Minor:

  • The related work discussion could be better aligned with the empirical comparison. GAS and CAPS are evaluated and introduced in the experimental section, but their relationship to the proposed method is not discussed in Section 2.
Questions for authors
  1. Could the authors clarify why C2IQL and BCRL are not included in the experiments? A comparison under the same filtered-data protocol would help assess the benefits of the proposed method relative to these closely related approaches.
  2. Could the authors provide more details on the experimental protocol, including training durations and checkpoint selection? Please also clarify how comparable training and evaluation conditions were adopted across the evaluated methods.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their careful reading and constructive feedback. We have provided detailed point-by-point responses and revised the manuscript accordingly. The main updates are:

  1. Motivation and compliant demonstrations (Reviewer pBSJ, W1/Q1 and follow-up). We expanded the motivation in Section 1 and added experiments retaining 25%, 50%, 75%, and 100% of the original budget-compliant trajectories on eight Point tasks. FIBA-Flow achieves mean normalized cost below one at every retention level. We also analyze the returns of compliant and non-compliant trajectories to help explain the observed performance differences (Appendix A.5.3, Tables 7–8).

  2. Adaptive budget guidance (Reviewer pBSJ, Q3). We clarified how the mixture of action distributions serves as the guide’s training target and how the guide outputs a single continuous budget scale during execution (Section 4.3).

  3. Training duration (Reviewer AHdh, Q1). We specified that FIBA-Flow is trained for 100,000 updates (Table 3).

  4. Finite-sample MMD estimator (Reviewer AHdh, Q4). We added the estimator used to train the guide, explicitly showing the exclusion of self-pairs among the four generated actions and the weights assigned to the guide target actions (Appendix A.3, Eq. (27)).

  5. Comparison with BCRL (Reviewer AD2J, W1/Q1). We added a comparison on all 38 tasks under the same filtering protocol and deployment budget of 10. FIBA-Flow satisfies the mean cost constraint on all 38 tasks, compared with 18 for BCRL (Appendix A.5.1, Table 5).

  6. Short-horizon cost prediction (Reviewer AD2J, W2; Reviewer pBSJ, Q2). We clarified the supervision used to train the short-horizon cost predictor and its relationship to the cost-to-go estimate (Section 4.1). We added an ablation replacing recorded short-horizon cost supervision with TD training. Across eight Point tasks, the two approaches achieve average NR/NC of 0.23/0.65 and 0.22/0.66, respectively (Appendix A.5.2, Table 6).

  7. Related work (Reviewer AD2J, minor comment). We expanded the discussion of GAS and CAPS and their relationship to FIBA-Flow (Section 2).

We welcome any further suggestions, including clarifications or results from our responses that the reviewers would find helpful to include in the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.