acceptodds
Under review as a conference paper at ICLR 2027

Generative Actor Critic

Abstract

Conventional Reinforcement Learning (RL) algorithms, typically focused on estimating or maximizing expected returns, face challenges when refining offline pretrained models with online experiences. This paper introduces Generative Actor Critic (GAC), a novel framework that decouples sequential decision-making by reframing as learning a generative model of the joint distribution over trajectories and returns, , and as performing versatile inference on this learned model. To operationalize GAC, we introduce a specific instantiation based on a latent variable model that features continuous latent plan vectors. We develop novel inference strategies for both , by optimizing latent plans to maximize expected returns, and , by sampling latent plans conditioned on dynamically adjusted target returns. Experiments on Gym-MuJoCo and Maze2D benchmarks demonstrate GAC's strong offline performance and significantly enhanced offline-to-online improvement compared to state-of-the-art methods, even in absence of step-wise rewards.

Primary area:
generative models
Keywords:
Sequential Decision Making, Generative Model, Reinforcment Learninig

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9MrmOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Generative Actor-Critic (GAC), a framework for offline and offline-to-online reinforcement learning that models the joint distribution p(τ,y)p(\tau, y). The authors introduce a latent variable zz and train the generative model by maximizing an ELBO. GAC uses a Bayesian inference structure to support both exploitation and exploration. The authors evaluate the framework on modified D4RL tasks with sparse rewards.

Critical strengths
  1. Clear presentation and derivation. The method is developed through a clear probabilistic formulation grounded in Bayesian inference. The derivations are easy to follow, and the experimental results and accompanying discussion deliver the main ideas effectively.

  2. The analyses support the intended behavior of the method. The distinction between maximizing expected return and conditioning on a high target return provides a convincing motivation for the exploitation objective. The comparison with fixed-target conditioning offers empirical support for this design. Figure 2 also illustrates the intended difference between exploitation and exploration: exploitation concentrates on high-return outcomes with lower variance, while exploration produces a broader return distribution. The improvement in the replay buffer’s return distribution across online fine-tuning stages further supports the usefulness of this approach. These analyses help explain how the inference objectives affect behavior, beyond reporting aggregate performance.

Critical weaknesses
  1. Limited experimental scope. Although GAC cannot directly exploit step-wise rewards, Tables 8 and 9 show that it can perform competitively using only trajectory returns against baselines with access to such feedback. However, the evaluation remains limited to D4RL MuJoCo and Maze2D. OGBench [1] offers a broader range of challenging tasks suitable for sparse reward evaluation. Experiments on tasks such as Scene and Puzzle, using sparse success rewards, are needed to establish whether benefits of GAC extend beyond the current benchmarks.

  2. Insufficient baseline coverage Addressing the optimistic bias problem does not by itself establish better performance. The authors should include representative diffusion planners [2][3], or justify their exclusion.

  3. Overstated performance claim. The SoTA claim in the discussion section is not sufficiently supported by the limited benchmarks and baseline coverage. It should be qualified to the evaluated tasks, included baselines, and trajectory-return-only setting.

[1] Park, Seohong, et al. "Ogbench: Benchmarking offline goal-conditioned rl." International Conference on Learning Representations. Vol. 2025. 2025.

[2] Ajay, Anurag, et al. "Is conditional generative modeling all you need for decision-making?." arXiv preprint arXiv:2211.15657 (2022).

[3] Dong, Zibin, et al. "Diffuserlite: Towards real-time diffusion planning." Advances in Neural Information Processing Systems 37 (2024): 122556-122583.

Questions for authors

Q1. Exploitation appears to require iterative latent optimization rather than amortized inference. Could the authors report the average number of optimization steps and wall-clock latency? How does this cost compare with the baselines, particularly when replanning is required?

Q2. The model assumes conditional independence between trajectories and returns given the latent plan, (y⊥τ∣z)(y \perp \tau \mid z). Under what conditions is this assumption unreasonable? Could the authors discuss how such violations affect return prediction and planning?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 9Mrm (1/2)

We thank the reviewer for a careful reading of the manuscript.

W1. Limited experimental scope.

We agree that a broader evaluation would strengthen the work. MuJoCo and Maze2D are nonetheless full-fledged benchmarks for high-dimensional control and for long-range planning, and they stress different axes: GAC is competitive on locomotion and decisively ahead on long-horizon planning under fully delayed reward. OGBench is designed for goal-conditioned RL, and its single-task variants of Scene and Puzzle are labeled with step-wise rewards [1], so evaluating them with trajectory returns only would require a new protocol without reference results. Beyond this scope, LPT has demonstrated the model class on Franka Kitchen (9-DoF multi-task manipulation) and Connect Four (discrete actions, stochastic opponent). The same latent-variable formulation with variational Bayes inference-time computation has also been applied outside control, to language modeling [2] and to mathematical reasoning [3]. In future work we will extend GAC to harder settings such as OGBench, visual control and multi-agent tasks. We will state the current empirical scope in the limitations.

W2. Insufficient baseline coverage.

Most recent methods fundamentally rely on dense step-wise rewards, which are unavailable in our outcome-based setting, and the cited diffusion planners are no exception. Diffuser and Decision Diffuser use cumulative step-wise rewards as their guidance or condition, and DiffuserLite optionally adds a value function learned by offline RL for sparse rewards ([4], Eqs. 9–10). On Maze2D, Diffuser inpaints the goal state, which "is identifiable by being the only state in the dataset with non-zero reward" ([5], §5.1). With trajectory returns only, these planners reduce to conditioning on a fixed target return.

While Decision Diffuser (DD) focuses on the conditional distribution p(τ∣y)p(\tau|y), GAC models the complete joint distribution p(τ,y)p(\tau,y). This distinction is crucial because it naturally entails modeling a critic p(y∣τ)p(y|\tau), which enables us to move beyond passively chasing a fixed target return (as in DD) to performing active inference—specifically, optimizing the expected return or sampling from a dynamically adjusted posterior. As demonstrated in Table 1, the capability enabled by such inference (GAC-E[y]\mathbb{E}[y]) consistently outperforms the fixed-target conditioning strategy (GAC-y∗y^*, akin to DD) on all nine datasets.

W3. Overstated performance claim.

We will qualify the claim to the evaluated tasks, the included baselines and the trajectory-return-only setting, in which GAC achieves the best results in Tables 1 and 2. We acknowledge that GAC may lag behind specialized Bellman-based methods when dense, step-wise rewards are available (Tables 8 and 9). Our framework is explicitly designed for the "return-only" setting to bypass the scalability bottleneck of expert-driven reward engineering. Consequently, the performance gap is an acceptable trade-off for broader applicability.

AuthorsOpenReview ↗

Response to Reviewer 9Mrm (2/2)

Q1. Optimization steps and latency.

At execution time GAC incurs a one-off planning cost, while per-step policy execution is identical to a model-free policy. Exploitation is solved once per episode at the initial state. It runs at most 300 gradient steps, with early stopping at a relative change of 10−510^{-5}, and each step is one forward and backward pass through the prior network and the return head only; the trajectory decoder is not involved. In practice it stops after 42 steps on average and takes 0.4 s for 100 start states on one A800 GPU. Each action then costs one forward pass of the decoder, as in Decision Transformer. By contrast, Diffuser and Decision Diffuser replan at every environment step in a receding-horizon loop, with 20 and 100 denoising passes per plan in their original settings [5, 6], and [4] reports their decision frequencies as 1.3 Hz and 0.4 Hz on D4RL with an RTX 3090. Over a 1000-step episode, GAC therefore needs around 50 passes of a small network for planning, against 20,000 and 100,000 denoising passes.

In Fig. 5, replanning is triggered at waypoint BB. We use the executed partial trajectory as a condition to re-run the inference optimization (Eq. 3), which solves for a new latent plan that maximizes the expected return while adhering to the history constraint. We warm start from the previous plan, which is closer to the optimal than an initial random guess. Unlike exploitation and exploration, each replanning step also requires a forward and backward pass through the trajectory decoder. One replanning call takes about 2.1 s, and replanning is invoked once per episode in this experiment.

Q2. When is y⊥τ∣zy\perp\tau|z unreasonable?

As a modeling assumption it is mild, because the latent plan is learned and can absorb whatever in τ\tau predicts yy. The assumption applies only to the conditional pγ(y∣zy)p_\gamma(y|z_y) given a specific plan zyz_y. We intend for this local distribution to be unimodal: once a plan is fixed, its return outcome should be predictable. The multi-modality of returns arises from the latent plans themselves, since p(y∣τ)=∫pγ(y∣zy) p(zy∣τ) dzyp(y|\tau)=\int p_\gamma(y|z_y)\,p(z_y|\tau)\,dz_y can be very expressive (§3.2).

Precisely because zz can absorb any such information, the assumption becomes problematic for planning in stochastic environments. The decoder models both actions and next states given zz (§3.2), so zz can encode transition outcomes the agent does not control, and maximizing E[y∣z]\mathbb{E}[y|z] may then select a plan that counts on favorable transitions; the realized return then falls short of the prediction [7, 8]. GAC removes one such source by conditioning the prior on s0s_0 rather than generating s0s_0 from the plan (§3.2), and the evaluated D4RL tasks have deterministic dynamics given s0s_0. Closed-loop replanning (Eq. 3) will alleviate this by further re-infering the plan under the likelihood of the executed prefix, including its observed transitions, which raises the success rate from 15.4% to 37.0% in Fig. 5. On hopper-medium, Fig. 4 shows that achieved returns track the target returns and that predicted returns track the realized ones.

[1] Park et al., OGBench: Benchmarking Offline Goal-Conditioned RL, ICLR 2025.

[2] Kong et al., Latent Thought Models with Variational Bayes Inference-Time Computation, ICML 2025.

[3] Kong et al., Inference-Time Rethinking with Latent Thought Vectors for Math Reasoning, arXiv:2602.06584, 2026.

[4] Dong et al., DiffuserLite: Towards Real-time Diffusion Planning, NeurIPS 2024.

[5] Janner et al., Planning with Diffusion for Flexible Behavior Synthesis, ICML 2022.

[6] Ajay et al., Is Conditional Generative Modeling All You Need for Decision-Making?, ICLR 2023.

[7] Paster et al., You Can't Count on Luck: Why Decision Transformers and RvS Fail in Stochastic Environments, NeurIPS 2022.

[8] Yang et al., Dichotomy of Control: Separating What You Can Control from What You Cannot, ICLR 2023.

Thank you for the detailed responses, which have addressed many of my concerns. However, the limited experimental scope remains a concern. Single-task variants of OGBench could be evaluated using aggregated trajectory returns, as done for the current benchmarks, and would help broaden the evaluation.

I will therefore maintain my score, although I would not object to acceptance.

Reviewer_swonOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Generative Actor Critic (GAC), a latent-variable framework that models the joint distribution of trajectories and final returns and then uses different inference procedures for exploitation and exploration. The main idea is to learn latent plans (z) that capture trajectory-level behavior, use predicted expected return for exploitation, and perform online exploration by sampling plans conditioned on optimistic target returns slightly above the upper tail of the replay buffer. The setting is interesting because the method only receives final trajectory returns rather than step-wise rewards. The empirical results are also potentially quite strong, especially on Maze2D.

Critical strengths
  1. The final-return-only problem setting is meaningful, and some of the empirical improvements are large. I think the Maze2D results in particular are difficult to dismiss as small benchmark noise. The offline-to-online improvements are substantial, and the method is operating without step-wise reward information. This makes the approach worth investigating further.

  2. The exploitation/exploration decomposition is conceptually appealing. More specifically, using one learned generative model for both expected-return optimization and optimistic exploration is a clean idea. I also appreciated that the paper goes beyond aggregate returns and includes analyses of latent structure, trajectory generation, and a proof-of-concept replanning experiment.

Critical weaknesses
  1. I could not reconcile the exploitation objective in the main text with the procedure in Algorithm 3: Eq. (2) defines exploitation as optimization over a distribution (qϕ(z)q_\phi(z)): DKL(qϕ(z)∥pα(z)).D_{\mathrm{KL}}(q_\phi(z)\Vert p_\alpha(z)). The KL term seems important: it explicitly regularizes the inferred plans toward the learned latent prior. The paper subsequently says that Eq. (2) is used for test-time exploitation.

However, Algorithm 3 describes exploitation as $ z^\star=\arg\max_z \mathbb E[y\mid z], $ using direct gradient ascent initialized from the prior, without the KL term. Table 3 describes exploitation in the same way.

These are not the same optimization problem, so I am unsure which one actually produced the reported GAC-(E[y]) numbers. This is important because those numbers constitute much of the paper's main empirical evidence. If Algorithm 3 is the actual implementation, this may ultimately be a presentation/specification issue rather than a problem with the method, but I think it needs to be resolved clearly.

I also found the motivating discussion around max⁡τE[y∣τ]\max_\tau E[y\mid\tau] somewhat confusing. Since yy is defined as the return associated with the complete trajectory τ\tau conditioning on a fully realized τ\tau seems to largely determine yy. The practically meaningful object appears to be closer to max⁡zE[y∣z]\max_zE[y\mid z], where zz represents a controllable plan while environmental stochasticity remains. I do not think this invalidates the implemented idea, but the decision-theoretic motivation should be revised.

  1. The empirical results are promising, but the uncertainty reported in the main tables does not appear to measure variability across independent training runs.

Tables 1 and 2 state that the means and standard deviations for GAC are computed from 100 evaluation trajectories. I could not find a corresponding aggregation over multiple independently trained GAC models in the submitted manuscript. The appendix mentions the cost of running “one D4RL task, one seed,” but does not report multi-seed training statistics for the headline comparisons.

For the large Maze2D gains, I would especially like to know whether they reproduce across independent training seeds. Variability over 100 rollouts from one learned checkpoint is useful, but it does not tell us whether the optimization/training procedure itself is stable.

The online exploration rule also introduces manually selected q and Δy\Delta y, and the appendix shows that these values sometimes vary across stages and tasks. I would like more detail on how these were selected and whether environment performance was involved in that process.

Finally, The contribution relative to LPT (Kong et al., 2024) is also difficult to isolate. LPT already provides latent trajectory-return modeling and planning through latent inference without step-wise rewards. GAC simultaneously changes the latent factorization, initial-state conditioning, prior architecture, variational optimization, exploitation rule, and online exploration mechanism. Without component ablations, it is difficult to determine whether the reported gains support the novel inference ideas specifically or the broader collection of modeling and optimization changes.

Overall, I like the direction of the paper. My main hesitation is that I could not fully determine the exact exploitation procedure underlying the reported results, and the current experimental reporting makes it difficult to assess how robust the large gains are across independently trained models.

Questions for authors
  1. Which exploitation procedure was actually used to produce the GAC-(E[y]) results: the KL-regularized distributional objective in Eq. (2), or the direct latent optimization in Algorithm 3? If it is Algorithm 3, how should Eq. (2) be interpreted relative to the implementation?

  2. How many independently trained GAC models were used for the results in Tables 1 and 2? Are the reported standard deviations purely over evaluation trajectories? Multi-seed training results for the main Maze2D experiments would substantially affect my confidence in the empirical claims.

  3. How were q and Δy\Delta y selected for online exploration, especially when their values change across stages? Were these values chosen using evaluation/environment returns?

  4. In Fig. 4, what inference procedure is used to obtain the latent plans for the critic-consistency plot? The training posterior is defined as qϕ(ϵ∣τ,y)q_\phi(\epsilon\mid\tau,y). Is ground-truth y used when inferring the latent plans whose predicted return is then compared against y, or is a trajectory-only inference procedure used?

  5. Could the authors provide at least one ablation separating the main modeling/training changes relative to LPT from the proposed exploitation/exploration inference rules?

3 comments
AuthorsOpenReview ↗

Response to Reviewer swon (1/2)

We sincerely thank the reviewer for their rigorous assessment and valuable suggestions.

Q1 / W1. Which exploitation procedure produced the GAC-E[y]\mathbb{E}[y] results?

The KL-regularized objective in Eq. 2. Algorithm 3 and Table 3 omit the KL term by mistake, and we will correct them. In practice, we optimize the mean and variance of q(ϵ)q(\epsilon), initialized from a prior sample, with AdamW for at most 300 steps, and execute the plan z=fα(s0,μ)z=f_\alpha(s_0,\mu). The same prior regularization is used in replanning (Eq. 3) and exploration (Eq. 4).

W1. Motivation via max⁡τE[y∣τ]\max_\tau\mathbb{E}[y|\tau].

Since a complete τ\tau largely determines yy, the decision-theoretic objective should be stated over the plan, max⁡zE[y∣z]\max_z\mathbb{E}[y|z], where zz is controllable while environmental stochasticity remains. We will revise the motivation in §3.1 accordingly.

Q2 / W2. How many independently trained models were used?

Tables 1 and 2 report one trained model per dataset. Our ±\pm is the dispersion of 100 rollouts within one run, a spread of outcomes rather than an uncertainty of the mean, and the corresponding standard error is a tenth of the printed value. We will explicitly document this convention in the captions. We also note that Maze2D deviations are intrinsic to the task. The outcomes are bimodal, the agent either reaching the goal with a high score or failing at close to zero, as the per-cell heatmaps in Fig. 3 show. A Table 1 entry averages over start cells, so a mixture of zeros and high scores makes a large deviation a consequence of a high mean rather than evidence of instability. For maze2d-medium, where the gain is largest, our hyperparameter study provides independently trained models at the context length used in the paper (M=16M=16) with different outer-loop steps NN. At Δy=100\Delta y=100, their GAC-p(y+)p(y^+) scores are 63.3, 55.2, 62.4 and 42.6 for N=1,5,10N=1, 5, 10 and 2020, where N=1N=1 is the model in Table 1. All are far above LPT (20.6). These runs differ in NN rather than in the seed.

Q3 / W2. How were qq and Δy\Delta y selected?

qq and Δy\Delta y are manually tuned on the environment returns of the exploration rollouts collected in each stage, not on the evaluation episodes. They are re-tuned at each stage, hence the changes across stages in Table 7. The architecture and optimization hyperparameters are selected by maximizing the ELBO with Optuna, without environment returns (Appendix B). While optimal Δy\Delta y varies by task, there is always a range of Δy\Delta y for each task that results in comparable performance. But we do agree that automating Δy\Delta y (similar to entropy tuning in SAC) is a promising direction for future work, and §5 already names the manual tuning of Δy\Delta y as a key limitation. The large Δy\Delta y for Maze2D is necessitated by the nature of this specific dataset, where the vast majority of trajectories yield 0 returns. A significant increment is required to push the exploration target beyond this dense, suboptimal mode to incentivize non-trivial pathfinding. The final GAC-p(y+)p(y^+) performance is:

Δy\Delta y15102050
hopper-m50.7±8.156.2±9.263.1±8.363.5±10.158.6±11.3
hopper-m-r37.2±22.745.1±21.562.0±22.359.1±19.452.5±24.8
Δy\Delta y10204080100
maze2d-medium-1.7±7.3-4.6±1.88.2±33.268.1±69.263.3±71.4
maze2d-large-2.0±1.90.6±7.035.3±30.727.4±42.139.5±42.3
AuthorsOpenReview ↗

Response to Reviewer swon (2/2)

Q4. Is the ground-truth yy used when inferring the latent plans in Fig. 4?

No. The latent plans are inferred from the trajectory alone, by maximizing the ELBO without the return term, and their averaged predicted return is compared with the ground-truth yy. The ground-truth yy only enters the opacity of the points through the ELBO, and we will clarify this in the caption.

Q5 / W2. Ablation separating the modeling changes from the inference rules.

We provide a single-module ablation on the Hopper task. To isolate the contribution of the model and training, we keep the objectives of GAC's inference queries fixed across all variants: "No Latent Split" (reverting to LPT-style coupled latent tokens), "No s0s_0" (removing initial state conditioning), and "Langevin" (replacing our gradient-based optimization of zz with LPT's Langevin sampling). The "LPT" column uses LPT's full architecture and training. Removing or replacing any one module increases the training loss and lowers GAC-E[y]\mathbb{E}[y]. The prior architecture (U-Net to Transformer) is not ablated separately; it changes together with the other components in the LPT column.

TaskMetricGACNo Latent SplitNo s0s_0LangevinLPT
hopper-mFinal loss ↓0.06480.07060.07250.13540.1564
GAC-E[y] ↑67.6±5.253.8±4.954.7±7.152.7±6.149.1±9.1
GAC-p(y⁺) ↑63.1±8.352.5±6.544.4±7.151.7±8.450.2±10.7
hopper-m-rFinal loss ↓0.11910.12070.12460.20490.2167
GAC-E[y] ↑83.4±6.671.6±16.658.4±22.545.6±12.963.3±13.5
GAC-p(y⁺) ↑62.0±22.363.7±20.833.1±28.645.2±24.050.8±17.8

Conversely, Table 1 holds the model fixed and varies the query. As demonstrated there, the exploitation query (GAC-E[y]\mathbb{E}[y]) consistently outperforms the fixed-target conditioning strategy (GAC-y∗y^*, as in LPT) on all nine datasets. For the online phase, Table 2 compares GAC with LPT under online fine-tuning: GAC's gains surpass LPT's on eight of nine datasets, reflecting the exploration query together with the model changes.

Thank you for the detailed response. Your clarifications and ablations address most of my concerns, specifically the component ablations which help distinguish the contributions beyond LPT. My remaining concern is training-seed variability. I understand that the reported standard deviations measure variation across evaluation trajectories, but this is different from variation across independently trained models. The additional Maze2D runs vary the number of outer-loop steps rather than the seed, so they don't directly establish training stability. Please consider reporting results with the same configurations as well.

In the light of your rebuttal, however, I move to weak accept. Good luck!

Reviewer_S3RxOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors address the challenge associated with refining offline pretrained models with online experiences To do this, the authors propose a generative actor critic model learns the joint distribution of trajectories and rewards With this model exploitation amounts to maximizing the expected return over trajectories, and exploration amounts to sampling from the posterior The authors claim comparable performance to existing methods on standard benchmarks, and superior performance in the offline-to-online setting

Critical strengths
  • The problem is likely of interest to the community
  • The approach appears novel
  • The empirical analysis shows improved performance in the offline-to-online setting
Critical weaknesses
  • The paper organization is poor. The preliminaries are mixed with related work, and the method is mixed with background and some training details and results interpretation. I find this to be significant, if the paper is difficult to understand the impact is limited.
  • The tone of the paper is a bit casual
  • I am unsure of the significance of the paper's contributions. While improved performance is important, I'm not sure that alone can justify acceptance. The interpretations of the framework and findings should be improved. While there are several baselines, an ablation study would be helpful.
Questions for authors
  • The variance reported for GAC in table 1 is quite high, why is that?
1 comment
AuthorsOpenReview ↗

Response to Reviewer S3Rx

We thank the reviewer for the detailed and critical feedback.

W1. Paper organization.

§2 is organized as a progression toward GAC rather than as a survey: the actor-critic paragraph sets up the expectation-centric baseline, distributional RL makes the critic generative, and sequence modeling makes the actor generative, so that GAC can be introduced as a synergy of the two. The full related work is in Appendix A. Likewise, §3.1 opens with the optimism bias because each inference query is introduced as an answer to it.

We agree that the boundaries can be made sharper and will revise accordingly. §2 will state the problem setting first, namely the step-wise reward rtr_t, the return-to-go GtG_t and the episode return yy, of which GAC requires only yy, and keep the prior methods only as the concepts GAC builds on. In §3, training details will move to the appendix, and the empirical remarks in §3.3 will move to §4.

W2. Casual tone.

We will revise casual expressions throughout the paper, such as "As we all know", "Obviously" and "Hopefully" in §3.

W3. Significance of the contributions and ablation study.

We will lay out the three contributions in §1:

  1. GAC Framework: decoupling decision modeling by reframing policy evaluation as learning a generative joint distribution p(τ,y)p(\tau, y), and policy improvement as latent space inference.
  2. Versatile Inference: two designated queries, gradient ascent on E[y∣z]\mathbb{E}[y|z] for exploitation and conditional sampling on p(y+)p(y^+) for distributional exploration, both derived from the same joint model.
  3. Offline-to-Online Learning: a pretrained model is refined online by maximizing the same ELBO. So adaptation introduces no new objective, no separate critic and no dense rewards—the online phase is that same training on a buffer the model extends itself.

Learning from final outcomes rather than step-wise process rewards aligns with emerging trends in RL-based reasoning and Large Language Models (e.g., DeepSeek-R1 [1]). In these domains, valid step-wise rewards are often unavailable, making the ability to learn solely from trajectory-level outcomes a critical capability.

Beyond aggregate performance, the results show two things. As demonstrated in Table 1, the exploitation query (GAC-E[y]\mathbb{E}[y]) consistently outperforms the fixed-target conditioning strategy (GAC-y∗y^*) on all nine datasets, which shows the effect of the optimism bias. Exploration is a planned shift of the return distribution rather than the byproduct of a reward bonus computed in hindsight from past visitation. This adaptive exploration continuously and automatically enriches the training distribution with high-quality, diverse experiences, which in return improves the generative models, as shown in the middle column of Fig. 2.

We provide a single-module ablation on the Hopper task. To isolate the contribution of the model and training, we keep the objectives of GAC's inference queries fixed across all variants: "No Latent Split" (reverting to LPT-style coupled latent tokens), "No s0s_0" (removing initial state conditioning), and "Langevin" (replacing our gradient-based optimization of zz with LPT's Langevin sampling). The "LPT" column uses LPT's full architecture and training. Removing or replacing any one module increases the training loss and lowers GAC-E[y]\mathbb{E}[y].

TaskMetricGACNo Latent SplitNo s0s_0LangevinLPT
hopper-mFinal loss ↓0.06480.07060.07250.13540.1564
GAC-E[y] ↑67.6±5.253.8±4.954.7±7.152.7±6.149.1±9.1
GAC-p(y⁺) ↑63.1±8.352.5±6.544.4±7.151.7±8.450.2±10.7
hopper-m-rFinal loss ↓0.11910.12070.12460.20490.2167
GAC-E[y] ↑83.4±6.671.6±16.658.4±22.545.6±12.963.3±13.5
GAC-p(y⁺) ↑62.0±22.363.7±20.833.1±28.645.2±24.050.8±17.8

Q. Why is the variance of GAC in Table 1 high?

Our ±\pm is the dispersion of 100 rollouts within one run, a spread of outcomes rather than an uncertainty of the mean, and the corresponding standard error is a tenth of the printed value. On eight of nine datasets the exploitation query has both the higher mean and the smaller deviation than the exploration query, so dispersion tracks the inference query, a test-time choice, rather than stability. We also note that Maze2D deviations are intrinsic to the task. The outcomes are bimodal, the agent either reaching the goal with a high score or failing at close to zero, as the per-cell heatmaps in Fig. 3 show. A Table 1 entry averages over start cells, so a mixture of zeros and high scores makes a large deviation a consequence of a high mean rather than evidence of instability.

[1] DeepSeek-AI, DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, arXiv:2501.12948, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.