acceptodds
Under review as a conference paper at ICLR 2027

Generative Value Classifier

Abstract

Q-learning is premised on learning a function to predict the value of a state-action pair, with the goal of using such a function to guide policy improvement. Most approaches learn this function from scratch using scalar or distributional return targets. This discriminative approach leaves another source of supervision for learning representations unused: the actions themselves. Rather than directly predict a return distribution , we ask how likely the state-action pair is under each possible return, . This perspective yields a critic based on Bayes' rule whose generative component is a return-conditioned policy, which can share the actor's architecture. Training this policy supervises the critic with every action rather than with returns alone, and a squared-error loss anchors the critic's Q-value to the bootstrapped target. We evaluate our approach on both offline and online RL tasks and show gains over leading discriminative methods, including distributional RL methods.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Generative Classifier, Value Learning, Flow Matching

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_xmueOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes an algorithm GVC, which aims to learn the value function implicitly based on learning a generative action model and return prior. The paper justifies this approach through an intuitive decomposition with Bayes rule. GVC is compared against other methods that use regression-based and distributional based approaches to learn the VF on various online, offline, and offline-to-online RL experiments.

Critical strengths
  1. Simplicity and strength. The algorithm is very simple (good thing!) and is widely applicable in a variety of settings. In general, it outperforms other approaches to value function learning that are common such as regression-based approaches and distributional RL. The simplicity suggests that it could be an adopted approach rather than just something that is interesting.
Critical weaknesses
  1. Unclear on cause of exact benefit. While it is great that the algorithm performs well, its not clear to me why the algorithm actually does better. The ablation in Section 5.4 does not seem to actually answer this, and it answers more of which components help the algorithm work well. For example, Figure 5(a) is not very informative since it seems natural that as the weight goes to zero, GVC would underperform since that is one of the critical pieces of the algorithm. 5(c) is interesting but also not entirely informative. If there are deeper reasons for the benefit such as how it relates to classification losses, the learned representations etc (see Questions below), that would be more beneficial. In other words, it seems that Q3 in Section 5 is incomplete.
Questions for authors
  1. While it seems clear that GVC outperforms distributional RL methods, it seems that the focus is more about the distributional part rather than the classification loss usage. For example, Stop Regressing: Training Value Functions via Classification for Scalable Deep RL (Farebrother et al. 2024) discuss the use of a classification loss for VF training. The distributional loss does have a classification-like component, but something like the Stop Regressing paper is suitable baseline in the experiments that I suggest the authors include. Given this, is there some relation between GVC and classification based methods, especially in terms of the type representations/solutions learned by each approach (see the paper's ablations)?
  2. The paper seems to have some relation to Upside-down RL (Reinforcement Learning Upside Down: Don’t Predict Rewards - Just Map Them to Actions (Schmidhuber et al. 2020)). Is there a connection and if so, I suggest the authors include it their discussion?

I generally like the paper. If the classification-based VF loss baseline and better understanding of why GVC is actually helping are included/addressed, I will increase my score.

3 comments

We thank the reviewer for their helpful questions and are glad to see their generally positive view on the work.

 

Unclear on cause of exact benefit...if there are deeper reasons for the benefit such as how it relates to classification losses, the learned representations etc...that would be more beneficial...

We agree this is an important question. Our primary hypothesis is that the benefit is largely due to the representation learning that the generative loss enables, whilst maintaining the same information provided by traditional discriminative losses (e.g., MSE on the target value).

This is supported by Figure 5(b), wherein "Uniform flow loss" (which effectively disables return conditioning) performs substantially better than "No flow loss", although worse than GVC (where the flow loss is properly weighted by the target return distribution). This is further supported by the ablation performed in response to Reviewer rVsa, where a standard Q-head on the same architecture performs similarly to "Uniform flow loss" (68.7 vs. 70.9 at 1M steps on Tool Hang) but well below GVC (85.6).

We note that other hypotheses which may provide complementary insight are (a) improved performance of generative classifiers in low-data regimes [1] and (b) improved performance of generative classifiers in handling distribution shifts [2].

We did not directly analyze these axes in this work, so we are careful not to claim these properties in comparison to discriminative methods, but we believe this is important future work.

Figure 5(a) was intended to demonstrate the sensitivity of the method to the flow-loss coefficient (λF\lambda_F) and to show that its tuning is relatively forgiving near the default value.

Overall, we agree that this is an important topic and that our current analysis of Q3 provides an incomplete view; we will expand this discussion in the revision.

[1]: Ng, Andrew, and Michael Jordan. "On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes." Advances in neural information processing systems 14 (2001).

[2]: Li, Alexander, Ananya Kumar, and Deepak Pathak. "Generative classifiers avoid shortcut solutions." International Conference on Learning Representations. Vol. 2025. 2025.

 

While it seems clear that GVC outperforms distributional RL methods, it seems that the focus is more about the distributional part rather than the classification loss usage...Given this, is there some relation between GVC and classification based methods, especially in terms of the type representations/solutions learned by each approach (see the paper's ablations)?

Thank you for the comments. We think these are important topics to discuss. In terms of the relation between our work and classification based methods (e.g., C51, Stop Regressing) there is certainly a close relationship as our work can be thought of as an "inverted" classifier.

One confounding factor to discuss is that Stop Regressing paper compared classification methods to MSE regression, but several of the benefits, such as improved performance with stochasticity in dynamics, and the greater expressivity, are also found more broadly in distributional RL methods which include more recent work such as the use of diffusion or flow models to predict returns.

We did not analyze the behavior under non-stationary targets as their work did, and since our method does not apply a similar cross-entropy loss for the primary p(a | s, z) term, we do not have reason to believe the same properties transfer. We note that our framework does allow for this — in principle one could use a similar cross-entropy loss instead of an MSE loss for the value-anchoring term which may provide these benefits, but we did not do so in our submitted work.

We initially opted to not include this baseline as its authors report improvements of at most 40% over than the more standard C51 method, but C51 itself gets relatively poor performance on the measured OGBench tasks, only beating out pure BC/IQL compared to all other methods.

Nevertheless, we appreciate the suggestion to strengthen the submission and plan to report results for the baseline within a few days. We will post a followup comment with the results of the baseline.

 

The paper seems to have some relation to Upside-down RL...Is there a connection and if so, I suggest the authors include it their discussion?

Thank you for the suggestion, Upside-down RL is highly relevant and we will include it in our related work.

It is an earlier work which pre-dates Decision Transformer (Chen et al., 2021), which we do cite as a relevant past work, but this is similarly important context for readers. Like Decision Transformer, it trains a policy conditioned on the return, but does not learn a value function and instead uses Monte Carlo returns. In contrast, GVC learns p(a | s, z) with TD-bootstrapped return targets and inverts it to obtain Q(s, a).

Thanks to the authors for their reply. The speculation on potential reasons for the benefit makes sense, and I agree going deeper in that direction may be a separate question. It would still be nice to include this as part of the discussion. I think the experiment done for reviewer rVsa was a nice addition, and I recommend the authors add that as part of the ablation.

I look forward to seeing the other baseline results soon.

I have updated my score in light of the response and the new experiment in the response to rVsa.

We thank the reviewer for the follow-up and for their constructive feedback. As suggested, we will add discussion on the topics in our reply above about the more specific reasons for gain, related work, and the additional Q-head ablation to our manuscript. We also provide the additional baseline below.

In running the "Stop Regressing" baseline (which we refer to as "HL-Gauss"), we discovered a bug in the C51 baseline: the cross-entropy loss was applied to the raw logits, without a log-softmax. This implementation comes from the public Value Flows codebase, which our OGBench experiments build on top of. After fixing this bug, we re-ran C51 on all seven domains and ran the HL-Gauss baseline.

The performance of C51 is substantially higher after the fix, so our earlier comment that C51 performs poorly in this setting was inaccurate. It is now on par with several distributional baselines but remains below a handful of other competitive baselines overall (Value Flows, BFN), as well as GVC, so our final conclusions are unchanged.

For HL-Gauss, we tune the smoothing width σ\sigma using the paper's suggested default of 0.75×δ0.75\times \delta (δ\delta is the bin width) and 0.5x and 1.5x this value (0.375×δ0.375\times\delta, 1.125×δ1.125\times \delta) per env, on the default task. The 5-task average for each env (8 seeds per task) is shown below and will be included in the paper.

 

EnvironmentC51HL-GaussValue FlowsBFNGVC
scene-sparse6478948698
puzzle3x3-sparse5877219798
puzzle4x43228273242
puzzle4x51919131718
puzzle4x6774512
cube-double4414696666
cube-triple3114610

 

We greatly appreciate the suggestion to run this baseline as it led to catching this error.

Reviewer_T9N9OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a reinforcement learning system based on a distributional estimate of returns and a best-of-N action selection scheme. The core idea is to decompose the distribution of returns with Bayes’ rule into two conditional distributions whose normalized product is anchored to Q-values. The algorithm is evaluated empirically on four simulated robotic tasks in the online setting and the OGBench for the offline and offline-to-online setting, giving comparisons with several baselines and some ablations.

Critical strengths

The paper does a decent job of explaining all the steps and details of the proposed algorithm. Similarly, the appendix gives a comprehensive description of the experiment setup.

The limitations section and compute cost experiment in the appendix shows an initiative to fairly evaluate the algorithm by considering an aspect other than improved performance over baselines.

Critical weaknesses

Errors in empirical design and analysis.

The paper states that the decomposed form of the return distribution empirically outperforms its equivalent joint model (L207), which seems to be the only justification for this algorithm’s development. Given its critical role, it was disappointing to see this thread is never picked up nor substantiated with empirical evidence.

The experiment section states three questions (L296) but then proceeds to show a series of benchmark results against several existing algorithms, avoiding to contextualize the results, or revisiting them to give answers. Organizing the experiment section to directly answer questions would better motivate the specific benchmarks or baselines used.

The analysis section is simply a summary of the plots and tables. It is hard to identify what insights we gain from the experiments. While the authors state the success of their method compared to the baselines, they do not give insight into why or how their algorithms outperforms the baselines (L355, L367). It is unclear what is learned from the ablations section (L441). This would have been a good place to circle back to the questions posed at the beginning and try to answer those questions supported by the evidence.

The experiment conditions are not controlled properly to yield meaningful results. Some baselines numbers are reused from prior papers while others are rerun, and I don’t believe the implementation differences are taken into account. The hyperparameters of all algorithms are differently selected based on an incomplete description in the appendix.

All the baselines are introduced in a single paragraph without much substance nor motivation for their inclusion. Some baselines are used as stand-ins for ablations of the algorithm (e.g., BFN as an ablation where only the critic is different) (L316) but this comparison is not possible due to uncontrolled experiment setup (see previous point). Other baselines, including C51, IQN, CODAC, Value Flow, BFN are introduced but omitted from online experiments (Figure 3) without justification.

The paper claims performance gains over baselines by comparing average of 3 runs (8 in offline experiments) which is not really possible given the complex performance distributions of RL algorithms. With so few seeds, there is not much confidence in the sample mean and the standard deviation; showing individual runs and softening the claims would be a more honest way to present the results.

The appendix includes tables of GVC hyperparameters and sweeps used for tuning some of the baselines (Tables 3,4,5). Several of the hypers have different values per-task and it is unclear how they are selected. In comparison, baselines have lists of sweeps but it is unclear what procedure is used and what final values are selected and whether the baselines are tuned per-task or across tasks. It is also unclear how much tuning budget is spent on baselines versus GVC given it has a lot more hyperparameters compared to the baselines.

It would be helpful if the experiment section text clarified the setup or results. For example, the authors mention using the same “training recipe as QC” (L346). It is unclear which baselines are also trained with it.

The introduction and discussion sections refer to challenges in learning value functions, suggesting that value learning is one of the goals of this paper. But neither the methodology nor the experiments measure the quality of the values learned. The actual pursued goal is finding optimal policies in MDPs. While value-based RL algorithms use a value function as an intermediate step for policy improvement, being clear about the goal and outcomes of research is very important. For example, “We have presented GVC, a method for value learning that …” (L478) is a false statement and can mislead the reader.

A final note on experiments. While I commend the authors for their discussion of the computational overhead of their algorithm (L478 and Appendix B), measuring wallclock time of different algorithms implemented in different codebases can be misleading, and not a clear proxy for compute cost. Also the experiments do not control for or discuss capacity (i.e., number of learnable parameters) of GVC versus the baselines which is indeed shown to be an underlying reason for many “algorithmic advances” in RL (for example, Multi-Task Reinforcement Learning Enables Parameter Scaling. McLean et al., 2025).

Missing related literature.

The core idea of the work, learning the distribution of actions conditioned on achieving a return, is directly related to the subfield of “planning as inference” but this connection is missed. Highlighting this body of work better places GVC within existing works and could even serve as motivation for its development. (See Compress and Control, Veness et al, 2014; Planning as inference, Botvinick & Toussaint, 2012; Planning by Probabilistic Inference, Attias 2003)

A key design choice in GVC is binning the returns. By binning the returns the loss no longer is driven by L2 but can be driven by a cross-entropy loss, which has been observed to be more robust in RL (possibly a distributional RL advantage too), but also see the work by Imani and White (2018, 2026) and the "Stop Regressing" paper (Farebrother et al., 2024).

More suggestions to improve the paper.

  • L17 "Training this policy": What policy? You mean treating the conditional distribution P(s, a | z) as a policy?

  • L27 The paper emphasises learning good value functions but that is not what the approach or the experiments measure. The paper is interested in finding the optimal policy in MDPs.

  • L29: Claims regarding MC and TD need citation. Especially the statement about stitching information across trajectories.

  • L55 "Our key insight ...": The insight is not really explained. Could I apply this insight to supervised learning and say "Standard losses fail to take advantage of a key source of supervision: the distribution of the instances themselves." But that makes no sense. Your explanation of the key insight should explain why you think p(a | s, z) is different and better. You recover the one from the other anyway, suggesting there's no more information in the "key insight".

  • L135 "where the Q-function estimates the discounted return of the policy given a state and action" is not the definition of off-policy RL.

  • L154 This formula has no definition of epsilon, lambda, B, or D! Furthermore, it has no citation to try to track any of it down.

  • Figure 1 and 2. Why is there both a Figure 1 and a Figure 2? One just has strictly more detail.

  • L199-206 Equations 6-7 is just the conditional Bayes formula. Written out like this makes it seem like it's doing more than that. And the two sentences after are just describing non-essential steps.

  • L252 "a single noise–time draw": LLM-sounding jargon made up on the fly. Define jargon before using it.

  • L258 "for example via best-of-N sampling": why only "for example? What did you actually do? This is the first moment it was clear to me that actions are continuous and the use of a flow model for p(a | s, z) is fitting a continuous distribution. If you had discrete actions, would you just throw away flow models for discrete logits?

  • L259 "shifted atoms" is undefined jargon.

  • L278 "policy extraction mechanism" is undefined jargon. As well as "implicit and explicit extraction methods".

  • L285 I find it crazy that you keep writing the flow-matching squared error as a "surrogate" for the actual log probability. The result is that it looks like you're doing something more complicated than, "We fit a policy to the best-of-N samples used in the replay buffer using flow matching." But when I get down to 289, I realize I do not understand which losses depend on which parameters. Is each loss (except anchor) only optimizing its associated theta, phi, psi parameter?

  • L306/L312 It is mentioned that the environments used for both online and offline experiments are sparse reward tasks. An explanation on how their method explores would have been valuable.

Questions for authors

No questions. Please refer to the weaknesses section for suggestions.

5 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/3)

We thank the reviewer for the detailed feedback. We address the main points below.

 

The paper states that the decomposed form of the return distribution empirically outperforms its equivalent joint model (L207), which seems to be the only justification for this algorithm’s development.

Our motivation for the factorization is the one in L205–206: it avoids generating the state and requires only p(a | s, z) and p(z | s).

We do not include our experiments for the theoretically equivalent joint model as we believe the optimal design for this would require substantial research which to our knowledge has not been previously done, whereas the standard discriminative modeling and the return-conditioned policies have seen substantial optimization. We believe such work is best suited for a standalone analysis. We will remove the mention of this finding.

 

The experiment conditions are not controlled properly to yield meaningful results. Some baselines numbers are reused from prior papers while others are rerun, and I don’t believe the implementation differences are taken into account. The hyperparameters of all algorithms are differently selected based on an incomplete description in the appendix.

We use the same training regime that has been used in several prior papers (e.g., Value Flows) and thus are able to re-use some of the baselines as mentioned in the experiments section. Our method uses these same base training hyperparameters including architecture, except as described to adapt the architecture to model p(a | s, z) with conditioning.

 

Other baselines, including C51, IQN, CODAC, Value Flow, BFN are introduced but omitted from online experiments (Figure 3) without justification.

We included baselines for each environment in which their original work, or common subsequent work, demonstrated the method on the environments used.

We included 11 Offline RL baselines, 7 Offline-to-Online RL baselines, and 6 Online RL baselines, comparing to the best performing baseline methods.

In particular, C51 and IQN are not competitive methods and thus not commonly benchmarked on online RL in cited works on RoboMimic. CODAC and Value Flows both only show results for Offline RL or Offline-to-online RL with single-step actions.

 

The paper claims performance gains over baselines by comparing average of 3 runs (8 in offline experiments) which is not really possible given the complex performance distributions of RL algorithms. With so few seeds, there is not much confidence in the sample mean and the standard deviation; showing individual runs and softening the claims would be a more honest way to present the results.

We use 3 seeds only for the non-ablation RoboMimic experiments as the variance is generally low, as demonstrated by the curves in Figure 3 (3 seeds) and Figure 5 (8 seeds, ablations). The margins between methods on the harder task of Tool Hang substantially differ far past any method variance. All ablations, including those on RoboMimic, use 8 seeds, as stated on L1039-1040. The non-ablation experiments follow standard practice in RL on RoboMimic in using 3 seeds [1, 2, 3].

All OGBench experiments (offline and offline-to-online) use 8 seeds as the OGBench environments do have higher variance.

[1]: Ren, Allen, et al. "Diffusion policy policy optimization." International Conference on Learning Representations. Vol. 2025. 2025.

[2]: Zhang, Tonghe, et al. "Reinflow: Fine-tuning flow matching policy with online reinforcement learning." Advances in Neural Information Processing Systems 38 (2026): 106282-106319.

[3]: Mandlekar, Ajay, et al. "What matters in learning from offline human demonstrations for robot manipulation." arXiv preprint arXiv:2108.03298 (2021).

 

It would be helpful if the experiment section text clarified the setup or results. For example, the authors mention using the same “training recipe as QC” (L346). It is unclear which baselines are also trained with it.

All RoboMimic experiments follow the QC training recipe and all OGBench experiments follow the Value Flows training recipe, including all baselines for each environment. The only exception to this is described in Appendix C.2 and applied to all methods including baselines, where we follow past work in using sparse rewards for two tasks. This was our intention of the wording on 346-348 but we will re-write it to be more explicit.

 

Our reply is continued in the next comment due to space.

AuthorsOpenReview ↗

Official Comment by Authors (2/3)

“We have presented GVC, a method for value learning that …” (L478) is a false statement and can mislead the reader....The paper emphasises learning good value functions but that is not what the approach or the experiments measure. The paper is interested in finding the optimal policy in MDPs.

To clarify, as we state in the paper, our method does not modify the actor algorithm and demonstrates substantial performance gains over baselines where the only change is the value function.

Our method proposes a novel critic formulation and trains it with TD bootstrapping. We believe this fits squarely within methods for value learning, even if the focus is on policy performance as a result of using the value function.

 

...measuring wallclock time of different algorithms implemented in different codebases can be misleading, and not a clear proxy for compute cost.

This is placed in the Appendix and is not a focus of the paper, but is placed as a reference point for readers. We also discuss the relative computational cost in terms of the number of forward passes in the Limitations section, to cover the time complexity instead of the raw time.

 

Also the experiments do not control for or discuss capacity (i.e., number of learnable parameters) of GVC versus the baselines

We control for capacity within our architecture in Section 5.4 (Figure 5(b)) when we ablate disabling the generative loss (“No flow loss”) and only keep an MSE regression loss to the scalar TD target and find a substantial performance loss.

Furthermore, we ablate disabling the return weighting (“Uniform Flow Loss”, Figure 5(b)), and in our reply to Reviewer rVsa, we further compare to a direct Q-head on top of the same architecture, finding that the action modeling loss (Eq 11) is helpful but proper weighting by the return further improves policy performance (68.7% vs. 85.6% on Tool Hang at 1M steps).

 

The core idea of the work, learning the distribution of actions conditioned on achieving a return, is directly related to the subfield of “planning as inference” but this connection is missed.

Thank you for pointing out this connection, we had missed Compress and Control (Veness et al., 2015) and the planning-as-inference literature (Attias, 2003; Botvinick & Toussaint, 2012) in our search.

We agree that this work is relevant and will include it in our related work. In particular, Compress and Control (Veness et al., 2015) estimates Q(s,a)Q(s,a) by applying Bayes’ rule to learned models of p(s∣z,a)p(s\mid z,a) and p(z∣a)p(z\mid a) trained with MC returns and is therefore a related prior work for GVC.

GVC uses a different factorization, p(a∣s,z)p(z∣s)p(a\mid s,z)p(z\mid s), which avoids modeling states, and combines flow-based action modeling with TD-bootstrapped targets and value anchoring for continuous control. We will clarify these distinctions and narrow our novelty claims accordingly.

 

L154 This formula has no definition of epsilon, lambda, B, or D! Furthermore, it has no citation to try to track any of it down.

Thank you for pointing this out to us. In Eq. (4), ϵ∼N(0,ID)\epsilon\sim\mathcal N(0,I_D) is Gaussian noise, λ∼U[0,1]\lambda\sim\mathcal U[0,1] is the flow time, and DD is the dimension of xx [1]. B\mathcal B contains the sampled (ϵ,λ)(\epsilon,\lambda) pairs used to estimate the squared error, with the same pairs reused across classes [2,3]. We will add these definitions to the paper.

Eq. (4) uses the conditional flow-matching squared error [1] as a classification score, analogous to the diffusion classifiers in [2,3]. We will add these definitions and citations alongside the equation.

[1] Lipman, Yaron, et al. “Flow Matching for Generative Modeling.” International Conference on Learning Representations, 2023.

[2] Li, Alexander C., et al. “Your Diffusion Model Is Secretly a Zero-Shot Classifier.” Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 2206–2217.

[3] Clark, Kevin, and Priyank Jaini. “Text-to-Image Diffusion Models Are Zero-Shot Classifiers.” Advances in Neural Information Processing Systems, vol. 36, 2023.

 

Our reply is continued in the next comment due to space.

AuthorsOpenReview ↗

Official Comment by Authors (3/3)

Why is there both a Figure 1 and a Figure 2?

Figure 1 is intended to help readers understand the method at a glance and in comparison to past methods. Figure 2 goes into more depth for readers who want more details. We are happy to merge these if the reviewer feels this would improve clarity for readers.

 

L252 "a single noise–time draw": LLM-sounding jargon made up on the fly. Define jargon before using it.

We will amend this to be “sample.” We went back and forth between “draw” and “sample” as “sample” is overloaded in the paper so we preferred to make a distinction. We accidentally mixed them in the submitted version.

 

L258 "for example via best-of-N sampling": why only "for example? What did you actually do? This is the first moment it was clear to me that actions are continuous and the use of a flow model for p(a | s, z) is fitting a continuous distribution. If you had discrete actions, would you just throw away flow models for discrete logits?

We say "for example" as GVC can be used with other forms of policy extraction such as direct-Q gradients (Appendix A.3), albeit with reduced performance on more complex tasks such as RoboMimic Tool Hang. All main experiments use best-of-N sampling (L278–280),

For discrete actions, we believe this would be reasonable to do. In unreported preliminary experiments, we found that discretizing the actions on these continuous control tasks performs substantially worse (in terms of policy performance).

 

L278 "policy extraction mechanism" is undefined jargon. As well as "implicit and explicit extraction methods".

The works [1, 2, 3, 4] use the term “policy extraction” and [1, 4] use “implicit”/”explicit”. We follow this terminology in our submission.

Policy extraction refers to how a policy is obtained from the learned critic.

Implicit refers to selecting from a set of sampled actions (e.g., best-of-N). Explicit refers to training an actor on the critic (e.g., via its action gradient). We will add these definitions in the revision.

[1] Dong, Perry, et al. "What Matters for Batch Online Reinforcement Learning in Robotics?" arXiv preprint arXiv:2505.08078 (2025).

[2] Park, Seohong, Qiyang Li, and Sergey Levine. "Flow Q-Learning." International Conference on Machine Learning. PMLR, 2025.

[3] Park, Seohong, et al. "Is Value Learning Really the Main Bottleneck in Offline RL?" Advances in Neural Information Processing Systems 37 (2024): 79029-79056.

[4] Hansen-Estruch, Philippe, et al. "IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies." arXiv preprint arXiv:2304.10573 (2023).

 

L285 I find it crazy that you keep writing the flow-matching squared error as a "surrogate" for the actual log probability.

We use “surrogate” as the squared error is a tractable objective for training/scoring but is not the exact negative log-likelihood. Evaluating the latter requires integrating the flow and its divergence [1].

This terminology is also used in [2], which explicitly describes “using the flow matching loss as a surrogate for log-likelihood in the policy gradient”.

[1] Lipman, Yaron, et al. “Flow Matching for Generative Modeling.” The Eleventh International Conference on Learning Representations, 2023.

[2] McAllister, David, et al. “Flow Matching Policy Gradients.” The Fourteenth International Conference on Learning Representations, 2026.

 

But when I get down to 289, I realize I do not understand which losses depend on which parameters. Is each loss (except anchor) only optimizing its associated theta, phi, psi parameter?

Yes, this is correct. The flow loss updates θ\theta, the return-prior loss updates ψ\psi, and the actor loss updates ϕ\phi. The anchor updates both θ\theta and ψ\psi. The actor, return-conditioned action model, and return prior have separate parameters, and the TD targets are stopped-gradient.

Reviewer_rVsaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GVC constructs a critic by combining a return-conditioned action model with a state-conditioned return prior. It uses flow-matching residuals as likelihood-like energies over return bins and takes the resulting posterior mean as the Q-value. A squared-error anchor fits this mean to a bootstrapped target. A separate behavior-cloning flow proposes actions, which the critic ranks through best-of-N selection. The evaluation covers demonstration-assisted online RoboMimic, offline OGBench, and offline-to-online OGBench.

Critical strengths
  1. The empirical evidence covers several learning regimes and includes a close actor-matched baseline. Across six offline-to-online tasks, GVC reaches 98.8% average final success versus 88.7% for BFN, with results over eight seeds (Table 2). Offline results also improve over BFN on puzzle-4x4, from 32% to 42% domain-average success (Table 1). The gains are not universal, but they are substantial on several difficult tasks.

  2. The ablations provide evidence that generative supervision contributes beyond the architecture alone. On Tool Hang, removing the flow loss substantially reduces success, and uniform weighting across return bins recovers only part of the benefit (Figure 5). The anchor and shared-noise ablations further explain which components are necessary. Reporting the weaker effect on Square and the failed direct-gradient actor experiments is useful for understanding the method's limits (Appendix A).

Critical weaknesses
  1. The experiments do not fully isolate the value of the proposed generative scoring rule from auxiliary action modeling. Removing the flow loss within GVC shows that this supervision helps its architecture. It does not establish whether the return-conditioned energy and prior are preferable to an ordinary Q head trained with the same action-model auxiliary loss and comparable capacity. This distinction matters because the main conceptual contribution is the construction of the critic, while value anchoring already supplies direct scalar TD supervision (Eqs. (11)-(15)).

  2. The demonstrated benefit comes with a restricted policy-extraction setting and extra computation. Appendix A.3 reports that the tested direct-Q-gradient actors remain near zero success on Tool Hang, so the positive evidence currently concerns best-of-N ranking of behavior-policy proposals. Appendix B reports 15.6 ms per GVC update, comparable to Value Flows at 13.5 ms but several times the scalar-critic baselines. These are acknowledged limitations. However, the sample-efficiency curves alone do not establish when GVC is preferable under a fixed wall-clock or action-selection budget.

Questions for authors
  1. Is there a comparison with a standard scalar or categorical Q head using the same shared representation and return-conditioned action-model auxiliary loss? A focused control on Tool Hang would help determine whether the scoring construction adds value beyond the additional supervision.

  2. Using the existing best-of-N runs, can you report end-to-end training time and action-selection latency relative to BFN and Value Flows? Please clarify how the failed direct-gradient actor results constrain the intended use of GVC, rather than attributing them to an established cause without further evidence.

2 comments

We thank the reviewer for the thoughtful review and we appreciate the concrete suggestions. We respond to the main comments below.

 

The experiments do not fully isolate the value of the proposed generative scoring rule from auxiliary action modeling. Removing the flow loss within GVC shows that this supervision helps its architecture. It does not establish whether the return-conditioned energy and prior are preferable to an ordinary Q head trained with the same action-model auxiliary loss and comparable capacity....Is there a comparison with a standard scalar or categorical Q head using the same shared representation and return-conditioned action-model auxiliary loss? A focused control on Tool Hang would help determine whether the scoring construction adds value beyond the additional supervision.

We appreciate the suggestion to directly add a Q-head and bypass the likelihood computation to make the comparison cleaner. We test this with a variant of the ablation in Figure 5(b), but do not perform any likelihood computation and instead use a standard MSE loss on a head of the network, before the final action output.

We find that this performs similarly to the “Uniform Flow Loss” experiment, suggesting that the action loss provides representation learning benefits, but that this is insufficient on its own to match GVC. The results are below, showing success rate (%) on Tool Hang, with mean ± std over 8 seeds:

Method0 steps250k500k750k1M
GVC0.0 ± 0.036.1 ± 8.780.6 ± 8.682.0 ± 10.885.6 ± 6.1
Uniform flow loss0.0 ± 0.028.8 ± 9.645.8 ± 11.362.5 ± 6.270.9 ± 5.6
No flow loss0.0 ± 0.012.1 ± 4.926.0 ± 4.544.2 ± 12.551.9 ± 3.7
Scalar-Q Head0.0 ± 0.028.6 ± 9.742.8 ± 12.762.7 ± 6.768.7 ± 5.8

 

However, the sample-efficiency curves alone do not establish when GVC is preferable under a fixed wall-clock or action-selection budget....Using the existing best-of-N runs, can you report end-to-end training time and action-selection latency relative to BFN and Value Flows?

Thank you for the suggestion. We will report this in the next couple days in a new comment below. Our storage unexpectedly went offline before we were able to retrieve the metrics for this response.

We originally did not include this as this metric is dependent on the GPU/TPU type and thus we believe the per-step time provides a more useful metric to gauge relative performance.

 

The demonstrated benefit comes with a restricted policy-extraction setting and extra computation. Appendix A.3 reports that the tested direct-Q-gradient actors remain near zero success on Tool Hang, so the positive evidence currently concerns best-of-N ranking of behavior-policy proposals....These are acknowledged limitations...Please clarify how the failed direct-gradient actor results constrain the intended use of GVC, rather than attributing them to an established cause without further evidence.

Thank you for raising this. The direct-Q gradient results in Appendix A.3 are included as additional information for future practitioners, and are not a piece of our method. They indicate that the distilled actor performs reasonably well on some environments (Square, Puzzle-4x4) but falls short of GVC with Best-of-N on the harder Tool Hang task. We will add this to the discussion section as a direction for future work.

AuthorsOpenReview ↗

Official Comment by Authors (Followup Response)

We thank the reviewer for their patience and provide the end-to-end training time and action-selection latency as requested in their review.

 

Total training time for 1M steps on OGBench puzzle-4x4 (1 H100 GPU):

MethodTime (hours)Relative to BFN
BFN0.731.0×
Value Flows3.214.4×
GVC4.836.6×

 

Total training time for 1M steps on RoboMimic Tool Hang (1 H100 GPU):

MethodTime (hours)Relative to QC
QC6.321.0×
GVC23.173.7×

On RoboMimic, where BFN and Value Flows are not among our baselines, we compare to QC which shares the GVC training recipe apart from the critic (as does BFN on OGBench).

 

Inference latency (BS=1, 1 H100 GPU):

MethodMean (ms)
IQL0.55
ReBRAC0.52
FBRAC0.68
IFQL1.31
FQL0.48
BFN (N=8)0.92
C511.02
IQN1.10
CODAC0.53
Value Flows1.17
GVC1.18

 

We will add these tables to the Appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.