- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes an algorithm GVC, which aims to learn the value function implicitly based on learning a generative action model and return prior. The paper justifies this approach through an intuitive decomposition with Bayes rule. GVC is compared against other methods that use regression-based and distributional based approaches to learn the VF on various online, offline, and offline-to-online RL experiments.
Critical strengths
- Simplicity and strength. The algorithm is very simple (good thing!) and is widely applicable in a variety of settings. In general, it outperforms other approaches to value function learning that are common such as regression-based approaches and distributional RL. The simplicity suggests that it could be an adopted approach rather than just something that is interesting.
Critical weaknesses
- Unclear on cause of exact benefit. While it is great that the algorithm performs well, its not clear to me why the algorithm actually does better. The ablation in Section 5.4 does not seem to actually answer this, and it answers more of which components help the algorithm work well. For example, Figure 5(a) is not very informative since it seems natural that as the weight goes to zero, GVC would underperform since that is one of the critical pieces of the algorithm. 5(c) is interesting but also not entirely informative. If there are deeper reasons for the benefit such as how it relates to classification losses, the learned representations etc (see Questions below), that would be more beneficial. In other words, it seems that Q3 in Section 5 is incomplete.
Questions for authors
- While it seems clear that GVC outperforms distributional RL methods, it seems that the focus is more about the distributional part rather than the classification loss usage. For example, Stop Regressing: Training Value Functions via Classification for Scalable Deep RL (Farebrother et al. 2024) discuss the use of a classification loss for VF training. The distributional loss does have a classification-like component, but something like the Stop Regressing paper is suitable baseline in the experiments that I suggest the authors include. Given this, is there some relation between GVC and classification based methods, especially in terms of the type representations/solutions learned by each approach (see the paper's ablations)?
- The paper seems to have some relation to Upside-down RL (Reinforcement Learning Upside Down: Don’t Predict Rewards - Just Map Them to Actions (Schmidhuber et al. 2020)). Is there a connection and if so, I suggest the authors include it their discussion?
I generally like the paper. If the classification-based VF loss baseline and better understanding of why GVC is actually helping are included/addressed, I will increase my score.
3 comments
Thanks to the authors for their reply. The speculation on potential reasons for the benefit makes sense, and I agree going deeper in that direction may be a separate question. It would still be nice to include this as part of the discussion. I think the experiment done for reviewer rVsa was a nice addition, and I recommend the authors add that as part of the ablation.
I look forward to seeing the other baseline results soon.
I have updated my score in light of the response and the new experiment in the response to rVsa.
We thank the reviewer for the follow-up and for their constructive feedback. As suggested, we will add discussion on the topics in our reply above about the more specific reasons for gain, related work, and the additional Q-head ablation to our manuscript. We also provide the additional baseline below.
In running the "Stop Regressing" baseline (which we refer to as "HL-Gauss"), we discovered a bug in the C51 baseline: the cross-entropy loss was applied to the raw logits, without a log-softmax. This implementation comes from the public Value Flows codebase, which our OGBench experiments build on top of. After fixing this bug, we re-ran C51 on all seven domains and ran the HL-Gauss baseline.
The performance of C51 is substantially higher after the fix, so our earlier comment that C51 performs poorly in this setting was inaccurate. It is now on par with several distributional baselines but remains below a handful of other competitive baselines overall (Value Flows, BFN), as well as GVC, so our final conclusions are unchanged.
For HL-Gauss, we tune the smoothing width using the paper's suggested default of ( is the bin width) and 0.5x and 1.5x this value (, ) per env, on the default task. The 5-task average for each env (8 seeds per task) is shown below and will be included in the paper.
| Environment | C51 | HL-Gauss | Value Flows | BFN | GVC |
|---|---|---|---|---|---|
scene-sparse | 64 | 78 | 94 | 86 | 98 |
puzzle3x3-sparse | 58 | 77 | 21 | 97 | 98 |
puzzle4x4 | 32 | 28 | 27 | 32 | 42 |
puzzle4x5 | 19 | 19 | 13 | 17 | 18 |
puzzle4x6 | 7 | 7 | 4 | 5 | 12 |
cube-double | 44 | 14 | 69 | 66 | 66 |
cube-triple | 3 | 1 | 14 | 6 | 10 |
We greatly appreciate the suggestion to run this baseline as it led to catching this error.
We thank the reviewer for their helpful questions and are glad to see their generally positive view on the work.
We agree this is an important question. Our primary hypothesis is that the benefit is largely due to the representation learning that the generative loss enables, whilst maintaining the same information provided by traditional discriminative losses (e.g., MSE on the target value).
This is supported by Figure 5(b), wherein "Uniform flow loss" (which effectively disables return conditioning) performs substantially better than "No flow loss", although worse than GVC (where the flow loss is properly weighted by the target return distribution). This is further supported by the ablation performed in response to Reviewer rVsa, where a standard Q-head on the same architecture performs similarly to "Uniform flow loss" (68.7 vs. 70.9 at 1M steps on Tool Hang) but well below GVC (85.6).
We note that other hypotheses which may provide complementary insight are (a) improved performance of generative classifiers in low-data regimes [1] and (b) improved performance of generative classifiers in handling distribution shifts [2].
We did not directly analyze these axes in this work, so we are careful not to claim these properties in comparison to discriminative methods, but we believe this is important future work.
Figure 5(a) was intended to demonstrate the sensitivity of the method to the flow-loss coefficient (λF) and to show that its tuning is relatively forgiving near the default value.
Overall, we agree that this is an important topic and that our current analysis of Q3 provides an incomplete view; we will expand this discussion in the revision.
[1]: Ng, Andrew, and Michael Jordan. "On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes." Advances in neural information processing systems 14 (2001).
[2]: Li, Alexander, Ananya Kumar, and Deepak Pathak. "Generative classifiers avoid shortcut solutions." International Conference on Learning Representations. Vol. 2025. 2025.
Thank you for the comments. We think these are important topics to discuss. In terms of the relation between our work and classification based methods (e.g., C51, Stop Regressing) there is certainly a close relationship as our work can be thought of as an "inverted" classifier.
One confounding factor to discuss is that Stop Regressing paper compared classification methods to MSE regression, but several of the benefits, such as improved performance with stochasticity in dynamics, and the greater expressivity, are also found more broadly in distributional RL methods which include more recent work such as the use of diffusion or flow models to predict returns.
We did not analyze the behavior under non-stationary targets as their work did, and since our method does not apply a similar cross-entropy loss for the primary p(a | s, z) term, we do not have reason to believe the same properties transfer. We note that our framework does allow for this — in principle one could use a similar cross-entropy loss instead of an MSE loss for the value-anchoring term which may provide these benefits, but we did not do so in our submitted work.
We initially opted to not include this baseline as its authors report improvements of at most 40% over than the more standard C51 method, but C51 itself gets relatively poor performance on the measured OGBench tasks, only beating out pure BC/IQL compared to all other methods.
Nevertheless, we appreciate the suggestion to strengthen the submission and plan to report results for the baseline within a few days. We will post a followup comment with the results of the baseline.
Thank you for the suggestion, Upside-down RL is highly relevant and we will include it in our related work.
It is an earlier work which pre-dates Decision Transformer (Chen et al., 2021), which we do cite as a relevant past work, but this is similarly important context for readers. Like Decision Transformer, it trains a policy conditioned on the return, but does not learn a value function and instead uses Monte Carlo returns. In contrast, GVC learns p(a | s, z) with TD-bootstrapped return targets and inverts it to obtain Q(s, a).