- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Generative Actor-Critic (GAC), a framework for offline and offline-to-online reinforcement learning that models the joint distribution . The authors introduce a latent variable and train the generative model by maximizing an ELBO. GAC uses a Bayesian inference structure to support both exploitation and exploration. The authors evaluate the framework on modified D4RL tasks with sparse rewards.
Critical strengths
-
Clear presentation and derivation. The method is developed through a clear probabilistic formulation grounded in Bayesian inference. The derivations are easy to follow, and the experimental results and accompanying discussion deliver the main ideas effectively.
-
The analyses support the intended behavior of the method. The distinction between maximizing expected return and conditioning on a high target return provides a convincing motivation for the exploitation objective. The comparison with fixed-target conditioning offers empirical support for this design. Figure 2 also illustrates the intended difference between exploitation and exploration: exploitation concentrates on high-return outcomes with lower variance, while exploration produces a broader return distribution. The improvement in the replay buffer’s return distribution across online fine-tuning stages further supports the usefulness of this approach. These analyses help explain how the inference objectives affect behavior, beyond reporting aggregate performance.
Critical weaknesses
-
Limited experimental scope. Although GAC cannot directly exploit step-wise rewards, Tables 8 and 9 show that it can perform competitively using only trajectory returns against baselines with access to such feedback. However, the evaluation remains limited to D4RL MuJoCo and Maze2D. OGBench [1] offers a broader range of challenging tasks suitable for sparse reward evaluation. Experiments on tasks such as Scene and Puzzle, using sparse success rewards, are needed to establish whether benefits of GAC extend beyond the current benchmarks.
-
Insufficient baseline coverage Addressing the optimistic bias problem does not by itself establish better performance. The authors should include representative diffusion planners [2][3], or justify their exclusion.
-
Overstated performance claim. The SoTA claim in the discussion section is not sufficiently supported by the limited benchmarks and baseline coverage. It should be qualified to the evaluated tasks, included baselines, and trajectory-return-only setting.
[1] Park, Seohong, et al. "Ogbench: Benchmarking offline goal-conditioned rl." International Conference on Learning Representations. Vol. 2025. 2025.
[2] Ajay, Anurag, et al. "Is conditional generative modeling all you need for decision-making?." arXiv preprint arXiv:2211.15657 (2022).
[3] Dong, Zibin, et al. "Diffuserlite: Towards real-time diffusion planning." Advances in Neural Information Processing Systems 37 (2024): 122556-122583.
Questions for authors
Q1. Exploitation appears to require iterative latent optimization rather than amortized inference. Could the authors report the average number of optimization steps and wall-clock latency? How does this cost compare with the baselines, particularly when replanning is required?
Q2. The model assumes conditional independence between trajectories and returns given the latent plan, . Under what conditions is this assumption unreasonable? Could the authors discuss how such violations affect return prediction and planning?
3 comments
Response to Reviewer 9Mrm (2/2)
Q1. Optimization steps and latency.
At execution time GAC incurs a one-off planning cost, while per-step policy execution is identical to a model-free policy. Exploitation is solved once per episode at the initial state. It runs at most 300 gradient steps, with early stopping at a relative change of , and each step is one forward and backward pass through the prior network and the return head only; the trajectory decoder is not involved. In practice it stops after 42 steps on average and takes 0.4 s for 100 start states on one A800 GPU. Each action then costs one forward pass of the decoder, as in Decision Transformer. By contrast, Diffuser and Decision Diffuser replan at every environment step in a receding-horizon loop, with 20 and 100 denoising passes per plan in their original settings [5, 6], and [4] reports their decision frequencies as 1.3 Hz and 0.4 Hz on D4RL with an RTX 3090. Over a 1000-step episode, GAC therefore needs around 50 passes of a small network for planning, against 20,000 and 100,000 denoising passes.
In Fig. 5, replanning is triggered at waypoint . We use the executed partial trajectory as a condition to re-run the inference optimization (Eq. 3), which solves for a new latent plan that maximizes the expected return while adhering to the history constraint. We warm start from the previous plan, which is closer to the optimal than an initial random guess. Unlike exploitation and exploration, each replanning step also requires a forward and backward pass through the trajectory decoder. One replanning call takes about 2.1 s, and replanning is invoked once per episode in this experiment.
Q2. When is unreasonable?
As a modeling assumption it is mild, because the latent plan is learned and can absorb whatever in predicts . The assumption applies only to the conditional given a specific plan . We intend for this local distribution to be unimodal: once a plan is fixed, its return outcome should be predictable. The multi-modality of returns arises from the latent plans themselves, since can be very expressive (§3.2).
Precisely because can absorb any such information, the assumption becomes problematic for planning in stochastic environments. The decoder models both actions and next states given (§3.2), so can encode transition outcomes the agent does not control, and maximizing may then select a plan that counts on favorable transitions; the realized return then falls short of the prediction [7, 8]. GAC removes one such source by conditioning the prior on rather than generating from the plan (§3.2), and the evaluated D4RL tasks have deterministic dynamics given . Closed-loop replanning (Eq. 3) will alleviate this by further re-infering the plan under the likelihood of the executed prefix, including its observed transitions, which raises the success rate from 15.4% to 37.0% in Fig. 5. On hopper-medium, Fig. 4 shows that achieved returns track the target returns and that predicted returns track the realized ones.
[1] Park et al., OGBench: Benchmarking Offline Goal-Conditioned RL, ICLR 2025.
[2] Kong et al., Latent Thought Models with Variational Bayes Inference-Time Computation, ICML 2025.
[3] Kong et al., Inference-Time Rethinking with Latent Thought Vectors for Math Reasoning, arXiv:2602.06584, 2026.
[4] Dong et al., DiffuserLite: Towards Real-time Diffusion Planning, NeurIPS 2024.
[5] Janner et al., Planning with Diffusion for Flexible Behavior Synthesis, ICML 2022.
[6] Ajay et al., Is Conditional Generative Modeling All You Need for Decision-Making?, ICLR 2023.
[7] Paster et al., You Can't Count on Luck: Why Decision Transformers and RvS Fail in Stochastic Environments, NeurIPS 2022.
[8] Yang et al., Dichotomy of Control: Separating What You Can Control from What You Cannot, ICLR 2023.
Thank you for the detailed responses, which have addressed many of my concerns. However, the limited experimental scope remains a concern. Single-task variants of OGBench could be evaluated using aggregated trajectory returns, as done for the current benchmarks, and would help broaden the evaluation.
I will therefore maintain my score, although I would not object to acceptance.
Response to Reviewer 9Mrm (1/2)
We thank the reviewer for a careful reading of the manuscript.
W1. Limited experimental scope.
We agree that a broader evaluation would strengthen the work. MuJoCo and Maze2D are nonetheless full-fledged benchmarks for high-dimensional control and for long-range planning, and they stress different axes: GAC is competitive on locomotion and decisively ahead on long-horizon planning under fully delayed reward. OGBench is designed for goal-conditioned RL, and its single-task variants of Scene and Puzzle are labeled with step-wise rewards [1], so evaluating them with trajectory returns only would require a new protocol without reference results. Beyond this scope, LPT has demonstrated the model class on Franka Kitchen (9-DoF multi-task manipulation) and Connect Four (discrete actions, stochastic opponent). The same latent-variable formulation with variational Bayes inference-time computation has also been applied outside control, to language modeling [2] and to mathematical reasoning [3]. In future work we will extend GAC to harder settings such as OGBench, visual control and multi-agent tasks. We will state the current empirical scope in the limitations.
W2. Insufficient baseline coverage.
Most recent methods fundamentally rely on dense step-wise rewards, which are unavailable in our outcome-based setting, and the cited diffusion planners are no exception. Diffuser and Decision Diffuser use cumulative step-wise rewards as their guidance or condition, and DiffuserLite optionally adds a value function learned by offline RL for sparse rewards ([4], Eqs. 9–10). On Maze2D, Diffuser inpaints the goal state, which "is identifiable by being the only state in the dataset with non-zero reward" ([5], §5.1). With trajectory returns only, these planners reduce to conditioning on a fixed target return.
While Decision Diffuser (DD) focuses on the conditional distribution p(τ∣y), GAC models the complete joint distribution p(τ,y). This distinction is crucial because it naturally entails modeling a critic p(y∣τ), which enables us to move beyond passively chasing a fixed target return (as in DD) to performing active inference—specifically, optimizing the expected return or sampling from a dynamically adjusted posterior. As demonstrated in Table 1, the capability enabled by such inference (GAC-E[y]) consistently outperforms the fixed-target conditioning strategy (GAC-y∗, akin to DD) on all nine datasets.
W3. Overstated performance claim.
We will qualify the claim to the evaluated tasks, the included baselines and the trajectory-return-only setting, in which GAC achieves the best results in Tables 1 and 2. We acknowledge that GAC may lag behind specialized Bellman-based methods when dense, step-wise rewards are available (Tables 8 and 9). Our framework is explicitly designed for the "return-only" setting to bypass the scalability bottleneck of expert-driven reward engineering. Consequently, the performance gap is an acceptable trade-off for broader applicability.