acceptodds
Withdrawn from ICLR 2027

Bellman-Aware Conservative Intervention: Recursive Actor-Flow Selection for Offline Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Flow-based offline reinforcement learning can combine an expressive flow policy with an efficient one-step actor, yielding two candidate actions at each state. Selecting between them is not purely a local ranking problem because the value of a current action can depend on future actor–flow selections. We introduce Bellman-Aware Conservative Intervention (BACI), an offline actor-flow selection algorithm that keeps both action generators fixed and learns a dedicated decision critic. Its Bellman target recursively accounts for future actor–flow selections. At deployment, a flow proposal can replace the actor action when its estimated advantage exceeds a replacement cost, with accepted replacements executed with probability . Under explicit assumptions, we establish exact policy improvement within the restricted actor–flow policy class, identify conditions under which recursive selection captures gains missed by actor-policy evaluation, and characterize the effect of value-ranking errors under approximate deployment. Experiments with frozen actor–flow pairs trained by Guided Flow Policy (GFP) and Flow Actor-Critic (FAC) show average improvements over the corresponding pretrained actors on OGBench and D4RL. Fitted Q Evaluation (FQE) and original-critic controls show that recursive value learning provides additional benefits beyond direct ranking, with its advantage over FQE varying across benchmarks.

Primary area:
reinforcement learning
Keywords:
Offline Reinforcement Learning, Actor-Flow Selection, Policy Improvement, Recursive Value Learning, Flow Matching, Actor-Critic

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pouvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes an algorithm for existing flow-matching-based offline RL methods that selects between two actions: one from the flow-matching behavior model and one from the one-step policy distilled from it, together with Q-maximization. The method relies on several metrics and procedures, and the authors present a training procedure built on them along with a theoretical justification. Experiments are conducted on D4RL and OGBench.

Critical strengths
  • The authors provide sufficient theoretical grounding for the proposed method.
Critical weaknesses
  • I do not understand why both the flow policy's action and the one-step policy's action should be used. The flow policy is originally intended for behavior cloning and serves as a reference for training the one-step policy, so there does not seem to be a particular reason to use its action at deployment. Moreover, unlike the one-step policy, the flow policy requires Euler integration steps to generate an action, which incurs an inference-time cost. Given that the experimental results do not show a substantial performance improvement, I am not convinced that there is a good reason to consider both the flow policy and the one-step policy, as this method does.
  • The proposed method is complex and relies on various heuristics. Eq. 3 depends heavily on specific hyperparameters, and these parameters appear to have been carefully tuned. Although the authors provide theoretical justification, I do not think it constitutes sufficient grounds for these design choices.
Questions for authors
  • Compared with existing approaches that use only the one-step policy, how much does BACI increase inference time? Since BACI involves action generation by the flow policy and a critic-based selection step, it seems important to report how much the inference time increases.
  • The one-step policy is distilled from the flow-matching model while performing Q maximization based on rewards. Under these circumstances, when and why does the flow policy's action turn out better than the one-step policy's action? Table 4 shows that the flow policy performs very poorly on its own, so I am curious why it helps the final results.
  • What about applying this method in online fine-tuning rather than offline training? I wonder whether performance could improve if training proceeds together with the action-selection process during online fine-tuning.
  • How sensitive is the method to the hyperparameters used in the paper? The paper uses a variety of hyperparameters without particular explanation, so it seems important to address this.
Reviewer_GRiTOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Using pre-trained and frozen one step flow actor and BC flow policy resulting from GFP (guided flow policy) or FAC (Flow Actor-Critic), the paper proposed BACI which train an additional critic (named decision critic) whose the corresponding policy is the one that select at each state between the action generated by the one step actor and the action generated by the multi-step BC flow policy. Decision critics are trained using both actions from the one-step actor and the multi-step flow policy, a replacement cost and a pre-fix probability of playing the action generated by the multi-step flow policy given that it was selected. At inference, at each state, two actions are generated using the one step actor and the BC multi-step flow policy, then the decision critic is used to select one of them, if the action generated by the BC multi-step flow policy is choosing, it is applied with probability pp, else the action generated by the one-step actor is applied. The proposed approach is evaluated on OGBench and D4RL benchmarks.

Critical strengths

BACI is evaluated on a wide range of tasks (using both OGBench and D4RL), and compare with multiple baselines. On OGBench, BACI based on GFP pre-training improve the performances on most of the tasks. Similar result is observed on D4RL for BACI based on FAC pre-training. But the improvement remain moderate.

Critical weaknesses

First, novelty and motivation are missing: The paper is the follow up of GFP, FAC and even FQL. But those works have a commun purpose: How to train, in the offline RL setup, an expression policy which is fast at inference and also avoid backpropagation through time. So, using on top of the one-step actor, the multi-step flow policy and an addition selections critics break the aim of fast inference. Apart from improving performance on benchmarks, it is unclear why BACI is relevant.

Second, the approach is too heavy. We first have to train GFP or FAC, then use the trained actors (one-step actor and multi-step BC) to train the decisions critics. At inference query both actor and the decisions critics on those two actions. On top of the GFP or FAC's hyperparameters, the method introduce 5 additionnals hyperparameters (t, p, r_\max, c_0, \lambda_d). Even though, (r_\max, c_0, \lambda_d) are fix to (0.15,1.10,0.25)(0.15, 1.10, 0.25) for the main experiment, there is no justification or intuition about these chooses.

The writing style of the paper as well as its theoretical justification are difficult to follow and there are incorrect statement in the paper. For example: in row 101 and 102; the author wrote " zaz_a denotes the actor’s random input when the actor is stochastic. The flow policy starts from an independent sample zFz_F ". But the main idea of distillation behind GFP, FAC and FQL that z=zA=zFz=z_A=z_F. This make me wonder if the authors have understood the methods there are suppose to build their work on. Even when, I follow the authors logic (zAz_A and zFz_F independant); how does πA\pi_A depend on zFz_F as stated in row 138? The overall framing of the paper is difficult to follow and some part are irrelevant ; e.g - line 043 'Existing methods improve how the policy or value function is learned from the offline data' (that is the definition of offline RL), - Assumption 1) " transition kernel is Markov" that is include into the definition of MDP; the assumption 1) is meaningless; line 158 and 159, Proposition 2, Theoreme 2, etc. Lets take Theoreme 2; for equation (9) to have any meaning; we first have assume the the assumption 1 (more than eleven statements); then that

A=EK,s∼dπA[1d(s,K)≤rmax⁡,aA≠aF1min⁡j∈1,2qψj(s,aF)−min⁡j∈1,2qψj(s,aA)>c0+λdd(s,K)(QπA(s,aF)−QπA(s,aA)−c0−λdd(s,K))]>0,A = \mathbb{E}_{K, s \sim d^{\pi_{A}}}[\mathbb{1}\\{d(s, K) \leq r_{\max}, a_A \neq a_F\\}\mathbb{1}\\{\min_{j \in \\{1, 2\\}} q_{\psi_j}(s, a_{F}) - \min_{j \in \\{1, 2\\}} q_{\psi_j}(s, a_{A}) > c_0 + \lambda_d d(s, K)\\}(Q^{\pi_A}(s, a_F) - Q^{\pi_A}(s, a_A) - c_0 - \lambda_d d(s, K))] > 0,

(EK,s∼dπA[1d(s,K)≤rmax⁡,aA≠aF1min⁡j∈1,2qψj(s,aF)−min⁡j∈1,2qψj(s,aA)>c0+λdd(s,K)])×(sup⁡sEK[1d(s,K)≤rmax⁡,aA≠aF1min⁡j∈1,2qψj(s,aF)−min⁡j∈1,2qψj(s,aA)>c0+λdd(s,K)(QπA(s,aF)−QπA(s,aA)−c0−λdd(s,K))]−inf⁡sEK[1d(s,K)≤rmax⁡,aA≠aF1min⁡j∈1,2qψj(s,aF)−min⁡j∈1,2qψj(s,aA)>c0+λdd(s,K)(QπA(s,aF)−QπA(s,aA)−c0−λdd(s,K))])>0(\mathbb{E}_{K, s \sim d^{\pi_{A}}}[\mathbb{1}\\{d(s, K) \leq r_{\max}, a_A \neq a_F\\}\mathbb{1}\\{\min_{j \in \\{1, 2\\}} q_{\psi_j}(s, a_{F}) - \min_{j \in \\{1, 2\\}} q_{\psi_j}(s, a_{A}) > c_0 + \lambda_d d(s, K)\\}]) \times (\sup_{s}\mathbb{E}_{K}[\mathbb{1}\\{d(s, K) \leq r_{\max}, a_A \neq a_F\\}\mathbb{1}\\{\min_{j \in \\{1, 2\\}} q_{\psi_j}(s, a_{F}) - \min_{j \in \\{1, 2\\}} q_{\psi_j}(s, a_{A}) > c_0 + \lambda_d d(s, K)\\}(Q^{\pi_A}(s, a_F) - Q^{\pi_A}(s, a_A) - c_0 - \lambda_d d(s, K))] - \inf_{s}\mathbb{E}_{K}[\mathbb{1}\\{d(s, K) \leq r_{\max}, a_A \neq a_F\\}\mathbb{1}\\{\min_{j \in \\{1, 2\\}} q_{\psi_j}(s, a_{F}) - \min_{j \in \\{1, 2\\}} q_{\psi_j}(s, a_{A}) > c_0 + \lambda_d d(s, K)\\}(Q^{\pi_A}(s, a_F) - Q^{\pi_A}(s, a_A) - c_0 - \lambda_d d(s, K))]) > 0

and

0<p<(1−y)EK,s∼dπA[1d(s,K)≤rmax⁡,aA≠aF1min⁡j∈1,2qψj(s,aF)−min⁡j∈1,2qψj(s,aA)>c0+λdd(s,K)(QπA(s,aF)−QπA(s,aA)−c0−λdd(s,K))](EK,s∼dπA[1d(s,K)≤rmax⁡,aA≠aF1min⁡j∈1,2qψj(s,aF)−min⁡j∈1,2qψj(s,aA)>c0+λdd(s,K)])×(sup⁡sEK[1d(s,K)≤rmax⁡,aA≠aF1min⁡j∈1,2qψj(s,aF)−min⁡j∈1,2qψj(s,aA)>c0+λdd(s,K)(QπA(s,aF)−QπA(s,aA)−c0−λdd(s,K))]−inf⁡sEK[1d(s,K)≤rmax⁡,aA≠aF1min⁡j∈1,2qψj(s,aF)−min⁡j∈1,2qψj(s,aA)>c0+λdd(s,K)(QπA(s,aF)−QπA(s,aA)−c0−λdd(s,K))])0 < p < \frac{(1-y)\mathbb{E}_{K, s \sim d^{\pi_{A}}}[\mathbb{1}\\{d(s, K) \leq r_{\max}, a_A \neq a_F\\}\mathbb{1}\\{\min_{j \in \\{1, 2\\}} q_{\psi_j}(s, a_{F}) - \min_{j \in \\{1, 2\\}} q_{\psi_j}(s, a_{A}) > c_0 + \lambda_d d(s, K)\\}(Q^{\pi_A}(s, a_F) - Q^{\pi_A}(s, a_A) - c_0 - \lambda_d d(s, K))]}{(\mathbb{E}_{K, s \sim d^{\pi_{A}}}[\mathbb{1}\\{d(s, K) \leq r_{\max}, a_A \neq a_F\\}\mathbb{1}\\{\min_{j \in \\{1, 2\\}} q_{\psi_j}(s, a_{F}) - \min_{j \in \\{1, 2\\}} q_{\psi_j}(s, a_{A}) > c_0 + \lambda_d d(s, K)\\}]) \times (\sup_{s}\mathbb{E}_{K}[\mathbb{1}\\{d(s, K) \leq r_{\max}, a_A \neq a_F\\}\mathbb{1}\\{\min_{j \in \\{1, 2\\}} q_{\psi_j}(s, a_{F}) - \min_{j \in \\{1, 2\\}} q_{\psi_j}(s, a_{A}) > c_0 + \lambda_d d(s, K)\\}(Q^{\pi_A}(s, a_F) - Q^{\pi_A}(s, a_A) - c_0 - \lambda_d d(s, K))] - \inf_{s}\mathbb{E}_{K}[\mathbb{1}\\{d(s, K) \leq r_{\max}, a_A \neq a_F\\}\mathbb{1}\\{\min_{j \in \\{1, 2\\}} q_{\psi_j}(s, a_{F}) - \min_{j \in \\{1, 2\\}} q_{\psi_j}(s, a_{A}) > c_0 + \lambda_d d(s, K)\\}(Q^{\pi_A}(s, a_F) - Q^{\pi_A}(s, a_A) - c_0 - \lambda_d d(s, K))])}

No ablation on hyperparameters introduce (r_\max, c_0, \lambda_d), insuffisant related works.

Questions for authors
  1. Can the authors read and explain the result they obtain in Theorem 2 for example?
  2. See Weaknesses
Reviewer_6MYLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes Bellman-Aware Conservative Intervention (BACI), a policy-improvement method for pretrained actor-flow pairs. The paper proposes to fix both policies and learn an additional critic that accounts for the actor-flow selection. The paper provides policy-improvement results under explicit assumptions and evaluates the methods on OGBench and D4RL.

Critical strengths

The paper identifies a meaningful distinction between evaluating candidates under the original actor and evaluating them under repeated selection, although it is unclear why this switching is needed.

Critical weaknesses
  1. The motivation and justification of several central design choices are not clear enough.

    • The proposed problem is not well motivated. The paper claims that repeated actor-flow selection requires a different value function, but it does not explain why switching is needed in the first place. The paper mentions FQL, but in FQL the flow policy is only used as a reference, while the one-step policy is used for return maximization. It is unclear why switching back to the flow policy during deployment helps. For other methods, it is unclear when the flow actions provide useful information that the actor misses.
    • The switching cost is introduced abruptly, without enough intuition or motivation.
    • The selection probabilities for training and deployment are set to be different. It is unclear why not just use the same.
  2. The empirical results cannot support the proposed method convincingly enough. The main benchmark table compares against published results under different evaluation protocols, while the matched controls use only a single training seed. The matched controls are much more informative than the main benchmark table, but the results are quite close, and with just one seed, the robustness of the results is impossible to evaluate.

Questions for authors
  1. Why is switching the actor-flow policies needed?

  2. Can the authors explain the design of the switching cost?

  3. Why does the main configuration have different training and deployment probabilities? What if they use the same?

  4. Can the authors provide comparison results with matched actors across multiple training seeds, including uncertainty on their performance differences?

  5. How sensitive are the results to the replacement cost parameters and distance threshold, and what inference latency does BACI add relative to the original actor?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.