acceptodds
Under review as a conference paper at ICLR 2027

What Matters for Value Guidance of Flow Policies in Reinforcement Learning?

Abstract

Flow policies are increasingly used to model behavior policies in offline reinforcement learning, but their iterative generative process complicates value-based policy improvement. Value guidance offers an alternative that leaves the behavior policy parameters unchanged by steering the generation process with the gradient of a learned critic. Existing value guidance methods instantiate this idea in diverse ways, differing in the flow timestep at which guidance is applied, how many guidance steps are taken, and how the guidance signal is constructed from the critic. Because these factors are rarely varied or studied in isolation, it is unclear which of them drive performance. In this work, we first identify which of these factors matter through a controlled study that varies each independently, and then examine how the findings change when the critic is trained on actions produced by the guidance itself. We find that the guidance signal determines which flow timesteps admit effective guidance, while the benefit of additional guidance steps differs across tasks. A single guidance step often suffices in tasks where the behavior cloning (BC) policy already performs well, whereas more steps are needed to substantially improve performance in tasks where the BC policy rarely succeeds. We also find that when the critic learns from bootstrap actions generated by the guidance itself, the benefit of additional guidance steps saturates earlier, with only a few steps recovering most of the attainable improvement. Building on this, we show that applying only a few guidance steps at selected flow timesteps matches or exceeds guiding at every flow step while reducing inference cost by up to 63%, showing that these insights translate into practical and efficient value guidance. These findings suggest that guidance schedules and value learning should be designed together to exploit the iterative structure of flow policies for effective value guidance.

Primary area:
reinforcement learning
Keywords:
Reinforcement learning, Value guidance, Flow policy

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_r8EeOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper evaluates different methods for value guided flow policies. The paper starts with a pretrained BC policy and a critic and varies guiding time step, guiding iterations, and guiding signal. Experiments are conducted in OG bench and Robomimic. The main takeaways are: guidance is best applied after t=0.4, a small number of guidance steps is sufficient to boost performance, and different value guidance signals do differ. Furthermore, further training the value function on sampled actions improve performance.

Critical strengths
  • This paper is well written and easy to understand the main takeaways. The figures presented the results very clearly.
  • The experiments studies 3 main dimensions to a flow policy design which is pretty comprehensive. The environments are sufficiently diverse and representative to demonstrate the properties of different policies. The guidance budget is a clever mechanism to ensure fair comparisons across environments.
  • The results are informative and provide practical guidance for users.
Critical weaknesses
  • The only minor concern I have for the paper is hyperparameter sensitivity. The results suggest guidance scale need to be tuned to specific task (which is true for all guided sampling methods) but also one needs to pick a late enough guidance time step.
Questions for authors

NA

2 comments
  • “The results suggest one should tune both guidance scale and guidance timestep, which is hyperparameter sensitive”

We thank the reviewer for this important point. We agree that our findings do not remove the need to tune the guidance scale and timestep, which is shared by value guidance methods in general. Rather, we believe that our analysis and findings suggest how to substantially reduce the cost of setting them.

For example, since the effective guidance timesteps are defined by how each guidance signal is constructed (Section 4.2), practitioners can identify where to apply guidance from the construction of their signal rather than through a per-task search. Moreover, since our analysis shows that within this effective range what matters is applying a sufficient number of guidance steps rather than the exact timestep, the timestep within the range does not need to be tuned precisely. Building on this, Section 5.3 considers such a practical choice, where window schedules apply guidance at a fixed set of late timesteps, the same for every task and signal, instead of manually tuning them, and still largely retain the performance of dense guidance with fewer guidance steps. Nonetheless, we agree that reducing the sensitivity of value guidance to its hyperparameters is an important matter, and we leave it as future work. We will add this discussion to the revised version of the paper.


We thank the reviewer again for the positive assessment and the valuable feedback. We hope our response has addressed your concern, and please let us know if you have any further questions.

Thank the authors for their response. I have no other questions.

Reviewer_7pUsOpenReview ↗
Rating
4(+2)
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies value guidance for pretrained flow-based behavior policies in offline RL. It systematically analyzes how performance depends on the guidance signal, flow timestep, and number of guidance iterations, first with a fixed critic and then with guided bootstrapping, where the critic is adapted to the guided policy. Based on these findings, the authors propose sparse guidance schedules that achieve performance comparable to dense guidance with reduced inference cost. The approach is evaluated on OGBENCH and ROBOMIMIC manipulation tasks against several recent flow-policy offline RL methods.

Critical strengths

The paper studies an important and timely problem in a systematic manner. As flow policies become increasingly common in offline RL, understanding how value information should be incorporated into their iterative generation process is relevant. Rather than proposing another guidance variant, the paper provides a useful decomposition of existing approaches into guidance signal, timestep, and number of guidance iterations, allowing the individual contribution of these design choices to be studied in isolation.

The controlled experimental design is a particular strength. The initial experiments keep the pretrained BC policy and critic fixed while varying the guidance components independently. The introduction of a guidance budget additionally controls for the total magnitude of intervention, making comparisons between different numbers of guidance iterations substantially more meaningful. This makes the conclusions about when and how repeated guidance helps considerably easier to interpret than direct comparisons between complete algorithms.

The paper provides several practically useful insights rather than only reporting performance improvements. In particular, the experiments show that the appropriate guidance timestep depends strongly on how the guidance signal is constructed, that repeated refinement is substantially more useful when the underlying BC policy performs poorly, and that adapting the critic through guided bootstrapping can substantially reduce the number of guidance iterations required.

The analysis translates into a simple practical recommendation. The observation that guidance need not be applied throughout the entire flow trajectory motivates sparse guidance schedules, which retain performance close to dense guidance while substantially reducing inference cost in the reported experiments. This provides a meaningful connection between the diagnostic analysis and an actionable design choice.

The paper is generally well structured and transparent about the relationship to prior work. The distinction between the initial controlled study and the subsequent guided-bootstrapping setting is clear, and the paper explicitly attributes the investigated guidance signals to prior approaches rather than presenting them as novel. The progression from understanding individual factors to evaluating their interaction with value learning and finally constructing sparse schedules makes the empirical story coherent.

Critical weaknesses

The baseline comparison appears incomplete with respect to closely related flow-based offline RL methods. In particular, Q-Flow (Doo et al., 2026) seems especially relevant, as the paper directly builds on its noisy/intermediate-space value formulation as one of its primary guidance signals, while Q-Flow itself uses intermediate value gradients for flow-policy optimization. FQL (Park et al., 2025b) is another relevant established flow-based offline RL method that is discussed but not evaluated. Including these methods, or explaining why they are not appropriate comparisons, would make the empirical positioning considerably clearer.

The advantage of gradient-based guidance over simpler Best-of-N policy extraction is not sufficiently established. A straightforward alternative is to draw multiple actions from the pretrained BC flow and execute the one with the highest critic value. This keeps candidate actions directly sampled from the behavior model and allows the samples and critic evaluations to be batched. While IFQL with N=32 provides one such baseline, the paper does not investigate how this approach scales with N or compare it against sparse guidance under a comparable inference/computational budget. Such a comparison seems particularly important given that inference efficiency is one of the main motivations for the proposed sparse schedules.

The experiments supporting the general conclusions about guidance parameters are relatively narrow. The controlled analysis of guidance signal S, timestep T, and number of iterations I is performed on only three OGBENCH manipulation tasks and two ROBOMIMIC tasks. Although the final baseline comparison covers a substantially broader set of tasks, the paper's central conclusions about when, how often, and with which signal guidance should be applied are primarily derived from this smaller set. Additional evidence across more diverse tasks would strengthen the claim that these observations represent general principles of value guidance rather than properties of the particular manipulation settings considered.

Figure 4 does not fully repeat the controlled guidance study under guided bootstrapping. The reparameterized guidance signal S is dropped based on its weaker earlier performance, and the ROBOMIMIC grid additionally omits t=0.6 and K=10. While reducing the experimental grid is understandable, evaluating the complete set would more directly support the conclusion that the findings regarding S, T, and I remain unchanged once the critic is adapted to the guided policy.

Finally, the methodological novelty appears somewhat limited, with the main contribution being the empirical analysis and synthesis of existing guidance mechanisms.

Questions for authors

see weaknesses

6 comments
  • “The experiments supporting the general conclusions about guidance parameters are relatively narrow”

To address the concern regarding the narrow scope of empirical study, we expanded the controlled analysis in Section 4 from five to ten tasks, adding three additional OGBench tasks and two tasks from a new benchmark, OCBench [1]. Crucially, we observe that the observations on both two factors reported in the original submission hold on these additional tasks:

  1. Effective timestep (T): as in the original tasks, the noisy-space critic remains effective across all timesteps, whereas the one-step signals are effective only at later timesteps, with one-step reparameterized guidance falling to the level of the unguided BC policy at (Figure 1). This holds across the tasks because the effective range is set by how each signal is constructed and not confined to a specific task setting. We explicitly clarify this in Section 4.2 of the revised version of the paper.
  2. Required number of guidance steps (I): To better show the effect of iterations, we report below the success rates (%) with K=3 and K=10 guidance steps at t=0.8, where all three signals are effective. | Benchmark | Task | BC | NSC (K=3K=3 → K=10K=10) | QGF (K=3K=3 → K=10K=10) | Reparam. (K=3K=3 → K=10K=10) | K=3K=3 relative to K=10K=10 | |---|---|:-:|:-:|:-:|:-:|:-:| | OGBench | cube-double task2 | 0 | 51 → 82 | 56 → 82 | 32 → 71 | 46–67% | | | cube-double task4 (new) | 1 | 30 → 54 | 33 → 46 | 16 → 45 | 36–71% | | | cube-triple task2 | 0 | 25 → 44 | 29 → 46 | 22 → 40 | 53–64% | | | cube-triple task4 (new) | 0 | 16 → 47 | 16 → 38 | 9 → 28 | 32–40% | | | puzzle-4x4 task4 | 0 | 26 → 87 | 29 → 92 | 15 → 73 | 21–32% | | | puzzle-4x4 task2 (new) | 0 | 44 → 96 | 35 → 89 | 20 → 82 | 24–46% | | OCBench [1] | block-double (new) | 27 | 78 → 76 | 58 → 61 | 74 → 79 | 94–103% | | | chamber-easy (new) | 64 | 91 → 95 | 87 → 90 | 86 → 92 | 93–97% | | Robomimic | square | 52 | 78 → 78 | 76 → 76 | 80 → 82 | 98–100% | | | tool-hang | 40 | 84 → 86 | 80 → 80 | 88 → 89 | 98–100% |

The table shows that the observation on the number of iterations (I) holds on every task, including the new ones. On the six OGBench tasks, where the BC policy almost never succeeds, K=3K=3 stays well below K=10K=10 for every signal. On the four OCBench and Robomimic tasks, where the BC policy already succeeds in 27–64% of the episodes, K=3K=3 is within 6 points of K=10K=10. Beyond adding tasks, we further verify this relation between the quality of the base policy and the required number of iterations in a new analysis (Section 4.4, Figure 4). Specifically, we observe that improving the initial noise on the same OGBench tasks, while keeping the BC policy and the critic fixed, reduces the required number of iterations: starting the value-guided sampling from the noise whose BC action is the best of four candidates under the critic lets K=3K=3 recover 88–98% of the performance at K=10K=10, compared with 42–54% from a random noise. This indicates that the required number of iterations is governed by the quality of the BC samples rather than by the particular benchmark.

References:

[1] Seohong Park and Sergey Levine. Behavioral Cloning Mystery. arXiv:2610.07056, 2026.

  • “Section 5 doesn’t fully repeat the controlled guidance study under guided bootstrapping”

Following the suggestion, we completed the study under guided bootstrapping (Figure 5 in the revised version), and added one-step reparameterized guidance on all benchmarks, the missing t=0.6t = 0.6 and K=10K = 10 on Robomimic, and the full grid on the new OCBench tasks, so that the grid now exactly matches the controlled analysis of Section 4.

The results are primarily consistent with the original findings of Section 5: the effective timesteps of each signal remain as under EMaQ, guided bootstrapping improves the overall performance ceiling, and the performance saturates with fewer guidance steps there (Figure 6). This also holds for one-step reparameterized guidance, which gains the most on OGBench at K=10K = 10 (26 points over EMaQ). The only difference is that it requires slightly more iterations to saturate than the other two signals: at K=3K = 3 it reaches 76% of its performance at K=10K = 10, compared with about 90% for the noisy-space critic and one-step QGF. This is still a clear improvement over the EMaQ critic, under which the same signal reaches only 43% at K=3K = 3 and 72% at K=5K = 5, whereas under guided bootstrapping it reaches 76% at K=3K = 3 and 99% at K=5K = 5.

  • “The baseline comparison appears incomplete”

We thank the reviewer for this suggestion. We have added both FQL [1] and Q-Flow [2] as baselines in Table 2. Each method was tuned for every environment, and we provide the tuning protocol and the selected hyperparameters in the appendix B.7. The table below summarizes the average success rates (%) on each benchmark against dense guidance trained with guided bootstrapping.

BenchmarkOGBench (30 tasks)OCBench (5 tasks)Robomimic (2 tasks)
FQL [1]484043
Q-Flow [2]675341
Noisy-space critic + dense777687
One-step QGF + dense776979
One-step reparam. + dense737478

As reported, value guidance with guided bootstrapping outperforms both baselines on all three benchmarks. While Q-Flow is a strong baseline on the OGBench scene and puzzle environments (93–100%), it falls behind on OCBench and Robomimic, where keeping the BC policy fixed and improving it only through value guidance appears to be beneficial.

References:

[1] Seohong Park, Qiyang Li, and Sergey Levine. Flow Q-Learning. ICML, 2025.

[2] JaeHyeok Doo, Byeongguk Jeon, Seonghyeon Ye, Kimin Lee, and Minjoon Seo. Q-Flow: Stable and Expressive Reinforcement Learning with Flow-Based Policy. ICML, 2026.

  • “The advantage of gradient-based guidance over simpler Best-of-N policy extraction is not sufficiently established”

We thank the reviewer for this constructive feedback. We added the requested comparison as a new Section 5.4 (Figure 9). Specifically, we use the same critic trained with guided bootstrapping (localized guidance at t=0.8, K=10) and compare Best-of-N (BoN) policy extraction with N ∈ {2, 4, 8, 16, 32} against value guidance with K ∈ {1, 3, 5, 10}, with inference compute measured in GFLOPs per action. Full results for each guidance signal are in Figure 9, and here, we report their average success rate.

BenchmarkBoN at its best NBoN (N=32)Guidance (K=3)Guidance (K=5)
OGBench77% (N=16, 0.94 GFLOPs)76% (1.87 GFLOPs)74% (0.12 GFLOPs)87% (0.17 GFLOPs)
OCBench93% (N=8, 1.65 GFLOPs)91% (6.62 GFLOPs)84% (0.43 GFLOPs)87% (0.60 GFLOPs)
Robomimic80% (N=4, 0.055 GFLOPs)63% (0.44 GFLOPs)79% (0.028 GFLOPs)79% (0.040 GFLOPs)

On OGBench, gradient-based guidance approaches the best BoN performance with only three steps (74% versus 77%) and surpasses it with K=5K = 5 (87% versus 77%), while using substantially less compute. On OCBench and Robomimic, guidance with three steps comes within 1–9 points of BoN at its best N with 2–4× less compute. Notably, drawing more samples does not always help BoN and exhibits performance degradation especially on Robomimic beyond N=4 (80% at N=4 to 63% at N=32). The advantage of gradient-based guidance is the largest on OGBench, where the BC samples are rarely good and selecting among them is therefore limited, whereas guidance moves the action beyond them.

  • “Methodological novelty appears somewhat limited”

We would like to highlight that our contribution lies in uncovering how the design factors of value guidance interact, which has been largely overlooked. As one example, one-step reparameterized guidance is unreliable at early flow timesteps, where the one-step estimate of the final action is inaccurate and backpropagating through it yields poor guidance. One-step QGF was proposed to address this limitation by omitting the backpropagation through the one-step predictor. Interestingly, our analysis shows that when guidance is instead concentrated at late timesteps, one-step reparameterized guidance matches or even outperforms one-step QGF under the same critic: at t=0.8t = 0.8 with K=10K = 10, it reaches 86% against 76% on OCBench and 85% against 78% on Robomimic. A limitation attributed to the signal is thus resolved by where guidance is applied. More broadly, these interactions translate into concrete guidelines, which Reviewer r8Ee also found informative and practically useful for users. For instance, a few guidance steps at well-chosen timesteps match dense guidance at a fraction of its cost (Section 5.3). We believe that understanding how these design choices interact is a step toward more interpretable and effective value guidance.


We thank the reviewer again for the valuable feedback, which we believe has helped us substantially improve the paper through the added experiments and clarifications. We hope our responses have addressed your concerns, and please let us know if you have any further questions.

Reviewer_WjUyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies value guidance for behavior-cloned flow policies in offline reinforcement learning. It aims at answering the question of "WHAT MATTERS FOR VALUE GUIDANCE OF FLOW POLICIES" by examining how guidance signal, flow timestep, number of guidance iterations, and guidance budget affect performance.

Critical strengths

-The paper addresses a relevant problem in value guidance for flow-based policies. -It provides a systematic analysis of several key design factors. -The controlled study helps disentangles the effects of guidance signal, flow timestep, number of guidance iterations, and guidance budget while holding the underlying BC policy and critic fixed. -The study provides empirical information on how these factors interact, and suggests that effective guidance timestep depends on the choice of guidance signal (Fig. 1).

Critical weaknesses

-Section 4 results indicate that the success rate under different guidance vary significantly over flow timesteps. Noisy-space critic guidance is relatively robust across timesteps, while the reparam one-step signal is generally weaker. Inspecting Fig. 1, the results do not yield a relatively clear trend to answer “what matters?” Substantially more evidence is need to support such a strong claim. -Takeaway - “Within effective (T), the benefit of more iterations (I) depends on task properties, including the performance of the underlying BC policy." But what is task property? Is it possible to have a meaningful characterization given the heterogeneity of robotics tasks in real-life? -There seems a chicken-and-egg problem in section 5 where the observed performance gains cannot be cleanly attributed to the guidance configuration because changing the guidance configuration also changes the critic used to generate and evaluate the guided policy.

Questions for authors
  • What evidence supports the conclusion that the observed effects of guidance signal, timestep, and number of guidance iterations represent consistent and generalizable principles of value guidance, rather than task- or benchmark-specific empirical patterns?
  • How should the performance gains under guided bootstrapping be interpreted given the coupling between the critic and the guidance policy?
5 comments
  • “Figure 1 and the results do not yield a relatively clear trend to answer what matters”
  • “What evidence supports the conclusion that the observed effects represent consistent and generalizable principles of value guidnace?”

We thank the reviewer for this comment. The main trend in Figure 1 we intend to show is that successful value guidance requires both (1) an effective flow timestep for the chosen guidance signal and (2) a sufficient number of refinement steps within that timestep. We agree that the submitted version did not explain why the effective timestep range differs across guidance signals, or why the required number of refinement steps differs across benchmarks. This left open whether the findings are benchmark-specific. To address this, we revised Section 4 to explain and further validate both factors.

  1. Effective timestep (Section 4.2): We revised the explanation of Figure 1 to clarify why the guidance signals differ in their effective timestep ranges and why applying guidance outside these ranges collapses performance. Specifically, the noisy-space critic is directly trained to evaluate the noisy action at any flowstep, supporting a broad timestep range. However, both one-step signals rely on a one-step estimate of the final action, which incurs significant estimation error early in the flow timestep. One-step reparameterized guidance further backpropagates through this one-step estimate, whereas one-step QGF avoids this additional source of gradient error, helping explain their slightly different effective ranges.

  2. Require number of guidance steps (new Section 4.4, Figure 4): Figure 1 further shows that fewer refinement steps suffice when the base BC policy has nonzero success, whereas more steps are beneficial when it generally fails. However, this observation alone cannot separate the role of base policy performance from other benchmark differences. We therefore test whether improving the initial BC samples reduces the required refinement while keeping the task, base policy, and critic fixed. Specifically, on OGBench, we sample mm actions from the same BC policy, select the highest-valued action using the fixed critic, and initialize guidance from the noise that generated it. This improves the starting samples and allows us to test whether better samples require fewer refinement steps. 

SettingSuccess without guidanceK=3K=3 relative to K=10K=10K=5K=5 relative to K=10K=10
OGBench, random sample (m=1m=1)0%42–54%68–80%
OGBench, best of 2 (m=2m=2)7%69–81%89–95%
OGBench, best of 4 (m=4m=4)40%88–98%98–99%
OCBench [1], random sample (m=1m=1)45%92–99%99%
Robomimic, random sample (m=1m=1)50%99%96–100%

The table shows the success rate before guidance and the percentage of the K=10K = 10 success recovered with K=3K = 3, which quantifies the effect of guidance iterations. Success without guidance can be read as the performance of the base policy from which guidance starts. As the table shows, as this base performance improves, only K=3K = 3 guidance steps recover 88–98% of the success achieved with K=10K = 10, up from 42–54% across the three signals. These results indicate that a better initial noise, which yields better BC samples, requires fewer guidance steps, which validates the second key factor of our findings: the number of guidance steps needed is governed by the quality of the BC samples. This helps explain why fewer steps suffice on OCBench and Robomimic, and suggests that the trend reflects sample quality rather than benchmark-specific differences.

In addition to above clarification and experiments, we expanded our original analysis to include three additional OGBench tasks and two tasks from newly added OCBench benchmark, doubling the number of tasks used in analysis throughout both Section 4 and 5 (5 → 10). Importantly, the original trends remain consistent across these additional tasks, broadening the empirical support for our findings and the claim of the original submission.

References: [1] Seohong Park and Sergey Levine. Behavioral Cloning Mystery. arXiv:2610.07056, 2026.

  • “What is a task property referred in Takeaway 1?”

We agree that “task properties” was a vague term. In the revision, informed by the Section 4.4 analysis described above, we replace it with a more concrete characterization: how good the unguided BC policy’s actions already are, using BC success rate as an observable proxy. The analysis shows that fewer guidance steps are needed when guidance starts from better BC samples, even on the same task. The consistent trends across the additional OGBench and OCBench tasks, together with those observed in the original tasks, including Robomimic, further support this characterization beyond the originally evaluated tasks.

  • “Section 5 has a chicken-and-egg problem where the observed performance gains cannot be cleanly attributed to the guidance configuration”

We would like to clarify that Section 5 serves a different purpose. Section 4 isolates the effect of guidance configuration by holding the critic fixed and shared across configurations. Section 5 examines what happens to the findings and key factors identified in Section 4 when the critic is instead trained by bootstrapping from the values of guided actions at the next state, which we call guided bootstrapping. Its purpose is therefore to analyze how the observed trends change under this critic-training procedure, rather than to attribute performance gains to the guidance configuration alone.

  • “How should the performance gains under guided bootstrapping be interpreted?”

The gains with guided bootstrapping should primarily interpreted as improvements from training the critic with guided actions in the bootstrap targets. With EMaQ, the TD target is computed with Best-of-N actions of the BC policy, so the critic can only be as good as the BC samples allow, whereas guided bootstrapping uses value-guided actions. This interpretation explains where the gains appear. On OGBench, where the BC samples are rarely good enough for task completion, the coupling improves the performance at K=10K = 10 by 10-26 points (Figure 6). On OCBench and Robomimic, where the BC samples already provide good target actions, the gains are smaller or absent. We have clarified this interpretation in Section 5.2 of the revised version of the paper.


We thank the reviewer again for the valuable feedback, which we believe has helped us substantially improve the paper through the added experiments and clarifications. We hope our responses have addressed your concerns, and please let us know if you have any further questions.

Thank you again for your thoughtful review and constructive feedback. We hope our clarifications and additional experimental results have addressed the concerns raised in your review and will be helpful in your final assessment. Please let us know if there are any remaining questions or concerns.

Comments on the paper

1 comment

We sincerely thank all reviewers for their thoughtful and constructive feedback. In the updated version of the PDF, we have improved our empirical evaluation and analysis, and all changes are highlighted in red. We summarize the main updates below.

  1. Expanded analysis. In response to Reviewers WjUy and 7pUs, we expanded the analysis in Sections 4 and 5 from five to ten tasks. We added three OGBench tasks (cube-double task4, cube-triple task4, puzzle-4x4 task2) and two tasks from a new benchmark, OCBench [1] (block-double, chamber-easy). We revised the paper throughout to incorporate the new benchmark, and Section 5 now repeats the full grid of Section 4 under guided bootstrapping, including one-step reparameterized guidance on all benchmarks. Notably, we show that original observations hold consistently across all ten tasks.

  2. Clarified the key factors and added a new analysis (Sections 4.2 and 4.4). In response to Reviewers WjUy and 7pUs, we revised Section 4 to clarify the role of the value guidance (the effective range of guidance timesteps and the required number of guidance steps) based on our empirical results, and show that the observed findings are not specific to the benchmarks considered. In Section 4.2, we explain in more detail why the effective range of guidance timesteps differs across guidance signals. Additionally, in the new Section 4.4 (Figure 4), we add an analysis showing that the required number of guidance steps is governed by the quality of the BC samples from which guidance starts.

  3. New baselines (Table 2). In response to Reviewer 7pUs, we added FQL [2] and Q-Flow [3] as baselines and extended the comparison to OCBench. The specific tuning protocol, selected hyperparameters, and baseline descriptions are provided in Appendix B.7 and C.

  4. Comparison with Best-of-N sampling (Section 5.4, Figure 9). In response to Reviewer 7pUs, we compare value guidance with Best-of-N policy extraction under the same critic, measuring inference compute in GFLOPs per action.

We provide detailed responses to each reviewer below.

References:

[1] Seohong Park and Sergey Levine. Behavioral Cloning Mystery. arXiv:2610.07056, 2026.

[2] Seohong Park, Qiyang Li, and Sergey Levine. Flow Q-Learning. ICML, 2025.

[3] JaeHyeok Doo, Byeongguk Jeon, Seonghyeon Ye, Kimin Lee, and Minjoon Seo. Q-Flow: Stable and Expressive Reinforcement Learning with Flow-Based Policy. ICML, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.