acceptodds
Under review as a conference paper at ICLR 2027

Imagined Action Sequence Guidance for Constrained Policy Optimization in Goal-Conditioned Reinforcement Learning

Abstract

Goal-conditioned reinforcement learning trains a policy to guide an agent toward a desired goal.Sparse rewards make useful and blocked actions difficult to distinguish near obstacles.Continuation Guided Constrained Policy Optimization (CGCPO) is proposed to combine imagined sequence guidance with a separate comparison of current action values.Candidate and reference actions condition a generator learned from recorded trajectories, and a sequence critic estimates the returns of the generated action blocks.These blocks are evaluated without intermediate feedback, whereas the deployed policy selects a new action after every observation.An independently trained primitive critic therefore compares the actual candidate and reference actions followed by the same reference policy.The policy objective favors a larger sequence value gap and penalizes primitive value gaps below a chosen margin.The trained policy selects one action per observation during execution.Under fixed policies and a fixed generator, the associated Bellman evaluations converge to their unique fixed points. The candidate policy's optimality gap is bounded by the sequence and primitive estimation errors, the generated block discrepancy, the candidate advantage spread and the remaining improvement in the policy score.Experiments on Fetch and PointMaze show the highest late success among the compared methods on the larger maze layouts and gains over baselines on obstacle manipulation.

Primary area:
reinforcement learning
Keywords:
Goal-Conditioned Reinforcement Learning, Reinforcement Learning, planning, sparse reward

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xLfDOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This article proposes a method called Continuation Guided Constrained Policy Optimization (CGCPO) based on the fact that the current GCRL method cannot effectively distinguish between useful and blocked actions at obstacles. It combines imagined sequence guidance with a separate comparison of current action values. CGCPO adds a generator on ddpg+her to generate actions quickly, a sequence value to score the quickness of actions, and an action loss to improve the difference in sequence values of generated blocks under candidate and reference actions, and a penalty is applied when the one-step critic value is below the boundary τ\tau.

Critical strengths

The initial insight of this article is excellent. It identified the issue that sparse rewards in the presence of obstacles result in the same rewards for different actions, and proposed a constraint strategy optimization based on continuation guidance to learn useful actions while avoiding blocking actions. The writing is clear and coherent, the diagrams are easy to understand, and it thoroughly discusses the connections with existing methods. Moreover, it has achieved advanced results in the fetch and pointmaze environments.

Critical weaknesses
  1. Although the author's proposed method is reasonable and logical, the idea of generating mirror sequences/tracks and the values of the sequences is indeed already an existing work in RL and has been well studied. Therefore, I do not consider the method to be particularly novel. Moreover, the second term of the penalty for the action loss (Equation 5) for the one-step critic is also similar to the previous CQL idea.

  2. I wonder how the sequence generator can ensure the correctness of the sequence without using a dynamics model for learning? Moreover, the sequence branch (generator + sequence critic) is conceptually close to the n-step TD error, but the key contrast between the two is not provided by the authors. It may have real advantages, such as using an entire block as input to make the variance of the regression target smaller, but this requires experimental proof.

For example, the data in the paper:

1). In Table 1, on the obstacle Pick&Place, the last-five of pure sequence guidance (NoPenalty) is 0.360±0.022, which is exactly the same as TD4's 0.360±0.048. On CustomLarge, the two are 0.908 vs 0.864, with a difference of approximately 1.7 standard errors.

2). In Table 2, by replacing flow with CGCPO_Det, which is deterministic mean block regression, the average of six tasks is 0.843, which is almost the same as Cap20's 0.847. Generative modeling is not necessary.

3). In FullGrad, where the first action is treated as a policy action (this is exactly how the n-step critic handles the first action), it achieves the highest score of 0.776 on the obstacle Pick&Place, which is higher than the default generative score of Cap20 (0.668).

  1. The hyperparameters in Appendix A.1 are almost exactly the same as the default configuration of HER in the OpenAI Gym baselines: polyak 0.95, noise 0.2, random actions 0.3, re-labeling 4/5, 40 updates per cycle, observation clipping ±5, target clipping [−1/(1−γ), 0]. However, this configuration's actor loss has a default action quadratic penalty of action_l2=1.0, which is not included in Equation (5). If the baseline retains this term, the coverage gap may largely come from "actions being compressed by L2", rather than credit assignment. The authors need to clarify this.

  2. Theorem 1 has two issues.

    First, it fails to explain the role of punishment. From b=c+λ1+λ(2e1−c)+≥cb = c + \frac{\lambda}{1+\lambda}(2e_1 - c)_+ \ge c, it can be seen that when η\eta is fixed, λ=0\lambda = 0 gives the tightest bound. The punishment only adds terms and does not subtract them. However, in the experiment, the punishment is precisely the most crucial component (removing it causes 5 out of 6 tasks to decline).

    Second, it cannot distinguish CGCPO from TD4. The advantage of sequential branching is only reflected in 11−γH\frac{1}{1-\gamma^H} being less than 11−γ\frac{1}{1-\gamma}, which holds for any nn-step critic.

  3. The fairness and coverage of the experiments.

1). Antmaze only has the "large" series of tasks. I suggest adding more maze experiments to demonstrate the performance and contribution of CGCPO, such as JaxGCRL [1].

[1] Accelerating goal-conditioned reinforcement learning algorithms and research, ICLR, 2025.

2). The most relevant action chunk value methods (Q-chunking, DQC, Chunk-Guided Q-Learning) are only distinguished as "offline, single critic" in related works, and no experimental comparisons have been conducted. The author has transformed CQN-AS into an online HER version, indicating that offline/online status does not constitute a reason for incomparable results.

This sole sequence baseline is almost 0 on Push as seen in Figure 3(b), and the evaluation protocol is inconsistent in the two domains (Appendix A.1), making it difficult to be regarded as a strong baseline.

3). The most straightforward n-step DDPG+HER only appears in ablation studies and has not entered the main comparison.

4). H has never conducted ablation studies and has not reported the computational cost.

Questions for authors
  1. "TD4 critic + the same step penalty" (without using the generator) - how effective is this? If it is comparable to CGCPO, what are the irreplaceable advantages of the generator?
  2. Is the environment transfer random? If it is deterministic, executing the same action block in both open-loop and closed-loop will result in the same trajectory. "Open-loop mismatch" is actually a deviation from the strategy in terms of deviation from the strategy. If it is random, using the closed-loop recorded blocks to regress to the open-loop value will have selection bias, which is inconsistent with the setting of Proposition 1. How to handle this?
  3. Do DDPG + HER, CGCPO_Prim, and CGCPO include action L2 regularization? After removing it, how is the reachability of DDPG + HER on CustomLarge?
  4. How to handle the transfer at the end of the round that is less than H steps? If these starting points are excluded, does the step that truly reaches the end from the original goal never enter the training of the one-step critic?
  5. In Figure 4(a), Δ^1\hat{\Delta}_1 is approximately 1e-3, while in Figure 11(d) on the same task it is around 1. What are the reference strategies at these two locations? How is the obstacle version of the Fetch task defined?
1 comment

We thank Reviewer xLfD for calling our starting insight excellent: near obstacles, sparse rewards give useful and blocked actions the same reward, and imagined long-term value tells them apart for the current action.

1. Novelty and the source of the gain: W1, W2, Q1

Our contribution is where sequence values act: imagined long-term value guides a one-step policy's current action, kept sound by a one-step check.

  • Novelty and CQL, W1. The policy acts once per observation, and imagined sequences credit it without being executed. Unlike DQC, our two values meet only in the actor loss; unlike CQL, which penalizes critic values, our hinge acts in the actor loss and both critics keep plain Bellman losses.
  • TD4 + the same hinge, Q1. At the same hinge, our method leads TD4 by 0.029 [0.015, 0.043] across eight tasks, and the Sequence column tops both rows of Table R2: the generator lets the continuation respond to the candidate action, unlike replayed TD4 targets.

Table R2. Obstacle Pick-and-Place, last-five success, 5 seeds.

HingeOne-stepTD4Sequence
Off0.3600.3400.392
On0.3680.6200.660
  • Generator and FullGrad, W2. Imagining actions, not states, the generator needs no dynamics model: it imitates executed replay blocks. The gain is generator-agnostic: cap-20 six-task means are 0.843 for deterministic regression and 0.847 for flow. FullGrad and SuffixGrad both pin the current action a0a_0 and score it with its generated suffix, unlike TD4; its direct derivative carries the gain, 0.776 on obstacle Pick-and-Place with it and 0.040 without, while the default, at 0.668, keeps the higher six-task average in Table 2.

2. Theory and dynamics: W4, Q2

Our theory certifies the current action chosen by imagined long-term value under the one-step check.

  • Role of λ\lambda. Theorem 1 holds for any HH-step estimate; the new bound in Sec. 4.4 adds λ\lambda: for margin τ≥0\tau\ge0, Adv−≥−c+2λe11+λ\mathrm{Adv}^-\ge-\frac{c+2\lambda e_1}{1+\lambda} with c=η+2ϵH+2DH1−γHc=\eta+\frac{2\epsilon_H+2D_H}{1-\gamma^H} tightens with λ\lambda whenever c>2e1c\gt 2e_1, that is, when sequence error dominates. CGCPO and TD4 then differ through ϵH\epsilon_H and η\eta, compared at the same hinge in Tables G1 and R2.
  • Dynamics, Q2. Fetch and PointMaze transitions are deterministic, so each recorded block is an exact open-loop rollout, free of selection bias. What remains is the gap you identify, between generated continuations and the re-queried πθ\pi_\theta, which the one-step hinge checks; Sec. 4.4 states this setting.

3. Implementation details: W3, Q3–Q5

Our gains come from crediting the current action with imagined long-term value, on a backbone all methods share.

  • Action L2, W3 and Q3. DDPG+HER, Prim, TD4 and our method all use action L2 = 1, now in Eq. 5, so coverage gaps reflect credit assignment. Without it, the DDPG+HER action norm nearly quadruples while success and reach fall, with two of five seeds at 0.04.

Table R3. DDPG+HER on CustomLarge, 5 seeds.

MetricL2 = 1, defaultL2 = 0
Last-five success0.4760.400
Reach at 963 fixed starts0.4250.369
Initial action norm0.3651.413
  • Terminal blocks, Q4. The goal-reaching reward enters every block return covering it; that transition starts no one-step update, and HER at ratio 0.8 supplies successful targets. Fetch never terminates early, so all but its last 3 of 50 steps start a block.
  • Fig. 4a versus Fig. 11d, Q5. Fig. 4a compares with the EMA target actor, tracking πθ\pi_\theta at polyak 0.95, so gaps stay near 10−310^{-3}; Fig. 11d accumulates the gap to an unguided reference pushing into the wall, reaching about 1, as the captions now state.
  • Obstacle tasks, Q5. Pick-and-Place has elevated goals and a pillar 0.04 m wide topped at 0.64 m; Push locks the gripper and places a wall 0.28 m wide; success is within 0.05 m at the final step.

4. Baselines, horizon, and compute: W5

Our method draws long-term credit from imagined action sequences, so we vary the horizon and add chunk-value baselines.

Table R4. Horizon sweep and DQC, last-five success, 5 seeds.

MethodObstacle Pick-and-PlaceCustomLarge
Ours, H=2H=20.5840.688
Ours, H=4H=4, default0.6600.916
Ours, H=8H=80.5920.940
DQC, online HER0.3600.912
  • Horizon and baselines. Longer horizons help long mazes, and H=4H=4 is best near the obstacle. DQC replaces CQN-AS* as the main chunk baseline and trails us by 0.300 there; TD4 joins the main comparison, Sec. 2 covers Q-chunking, Chunk-Guided Q-Learning and JaxGCRL, and Table G1 spans six maze layouts.
  • Compute. Deployment runs only πθ\pi_\theta, so inference matches DDPG+HER; at H=4H=4, first-to-last evaluation takes 2.62 h on CustomLarge and 4.29 h on obstacle Pick-and-Place.
Reviewer_e2CgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes CGCPO for sparse-reward goal-conditioned RL, which is an improvement on DDPG+HER. A flow-matching model completes the action of an actor into a block that is scored by a chunked critic (effectively behavior cloning of 4-step action chunks from replay). A capped penalty becomes active when the action scores below the EMA target actor under the standard one-step critic. The paper evaluates CGCPO on 4 Fetch and 6 PointMaze tasks with ablations.

Critical strengths

The method clearly improves over DDPG+HER on the long-horizon mazes (Fig.3) annd over the closely related one-step variant CGCPO$_{Prim}$ (Fig.5). Long-horizon goal reaching with sparse rewards is a relevant problem. The ablations are extensive and honestly reported, which makes it possible to check which components carry the gains.

Critical weaknesses
  1. The core idea is hard to extract. Standard components are renamed instead of using known terms. The "reference policy" seems to be the DDPG target actor and the "primitive critic" the standard DDPG critic. "Imagined sequences" sounds like model-based imagination like with world models, but as no dynamics model is learned and the generator is behavior cloning of action chunks this is a mislead. The likely mechanism (faster propagation of the sparse reward through 4-step backups) is never clearly stated. The actor gradient (Eq.C.50), which in my opinion describes the method most clearly, is only in the appendix, while substantial parts of the main text is theory that adds little to understanding. As a result, it is difficult to judge what is actually new compared to DDPG+HER and action-chunked Q-learning.

  2. It is unclear whether the new components are needed at all. If the main benefit is that 4-step blocks pass the success signal back faster, a simple 4-step target without generator and sequence critic should already be enough. The paper's own ablation suggests exactly this for the maze environments. The simple variant CGCPO_TD4 almost matches CGCPO on CustomLarge (cf.Fig.8), but this is hidden in the appendix, while the coverage argument in Sec.5.3 uses only the one-step variant. A clear gain over CGCPO_TD4 only appears on obstacle Pick&Place (cf.Tab.1). The theory does not explain why the sequence critic or the penalty should help. If the gains mainly come from multi-step targets (a well-known technique) the contribution imho is much smaller than claimed.

  3. The claim that CGCPO "stays within the top three on all ten tasks" is a bit misleading. On the two AntMaze-layout tasks only three methods are compared, so top3 is guaranteed there. On Fetch, CGCPO is below EBP on both Pick&Place tasks. The main result therefore depends on an uneven choice of baselines per task rather than on a consistent comparison.

  4. With 5 seeds and standard errors around 0.05-0.09, many of the ablation differences are within noise. No significance tests or aggregate metrics (e.g., IQM with bootstrap CIs) are reported.

Questions for authors
  1. Is the main idea that the 4-step blocks pass the sparse success signal back faster? If so, how does CGCPO compare to the simple 4-step variant (CGCPO_TD4) on all ten tasks?

  2. How is the term "imagined" related to world models, given that no dynamics model is learned? Are the reference policy and the primitive critic the same as the DDPG target actor and critic, and is the generator in effect behavior cloning of action chunks from the replay buffer?

  3. How was the penalty cap of 2 chosen, given that a cap of 20 or no penalty change the CustomLarge results considerably?

  4. Can the two AntMaze-layout tasks be evaluated with the full set of baselines?

3 comments

We thank Reviewer e2Cg for recognizing clear gains over DDPG+HER on long-horizon mazes and extensive, honestly reported ablations; below we show where the long-term value of imagined action sequences adds to four-step targets in guiding the current action.

1. Terminology and mechanism, W1 and Q2

Our method imagines in action space: the long-term value of generated action sequences guides the current action, with no state prediction.

  • Yes on both counts. The reference policy is the DDPG EMA target actor with a Polyak coefficient of 0.95, the primitive critic is the one-step DDPG critic, and the generator is behavior cloning of four-action replay chunks using flow matching conditioned on the state ss, goal gg, and initial action a0a_0; the sequence critic is an action-chunk critic. Sec. 4.1 states this mapping and defines "imagined" as generated actions; no dynamics model is learned.
  • Mechanism. Sec. 1 now names both parts: HH-step guidance through the imagined block, and a one-step hinge against the target actor. The actor gradient and Algorithm 1, as described in the response to ifwN, move from Eq. C.50 into Sec. 4.3.

2. Beyond four-step targets, W2, Q1, and W4

Our main idea goes beyond faster propagation: imagined long-term value gives the current action an action-dependent gradient, and the one-step check decides where that gradient is trusted.

  • All ten tasks, as shown in Table G1. Our mean exceeds TD4 on all ten: the eight new-batch tasks plus plain Pick-and-Place, where our method achieves 0.912 against 0.880, and obstacle Push, where it achieves 0.832 against 0.580, in the submitted batch. Propagation lifts CustomLarge from 0.476 to 0.912, yet TD4 stays at 0.340 on obstacle Pick-and-Place against our 0.660.
  • Significance, W4. Paired bootstrap over matched seeds with 50,000 resamples and equal task weighting gives +0.063 with a confidence interval from 0.049 to 0.077 over TD4 and +0.029 with a confidence interval from 0.015 to 0.043 over TD4 + hinge, which our method beats on six tasks and ties on the two saturated ones.
  • The check needs imagined guidance, as shown in Table G2. With TD4 the hinge lifts obstacle Pick-and-Place from 0.340 to 0.620 but lowers all three large mazes; with sequence guidance it adds 0.268 near obstacles, and our method stays at or above both TD variants on every task.
  • Coverage. TD4 now enters the coverage argument: in Fig. 8, CustomLarge reach is 0.85 for ours, 0.83 for TD4 and NoPenalty, and 0.40 for Prim, with 29 cells improved and 13 lost against TD4; in Fig. 9 the gradient cosine averages 0.245 and is negative in 38 cells. Both figures move into Sec. 5.3.

3. Ranking, Large-layout baselines, and the cap, W3, Q3, and Q4

Long detours in the Large layouts separate the current action from the sparse goal reward, which is where imagined long-term value must carry the credit.

  • Ranking, W3. Sec. 5.1 replaces "top three on all ten tasks" with task-by-task comparisons and states that EBP leads the Pick-and-Place tasks.
  • Full baselines, Q4. Both layouts use AntMaze-Large walls with the point agent, with starts and goals from all free cells or eight designated cells; Large-all uses 5 seeds, 3 for CQN-AS* and TMD, Large-eight 3.

Table R1. Large layouts, last-five success, means over seeds.

MethodLarge-allLarge-eight
Ours0.6560.780
CQN-AS*0.6530.787
TD40.6520.753
TD4 + hinge0.6160.727
TMD0.3400.460
EBP0.3240.347
  • Reading. Our method and CQN-AS* hold the top two means, while EBP and TMD stay at or below 0.460. No external baseline is strong in both domains: CQN-AS* stays below 0.15 on Push and obstacle Pick-and-Place, as shown in submitted Fig. 3, and DQC reaches 0.360 on obstacle Pick-and-Place. The submitted Large runs, reported separately, give GCSL 0.360 on Large-all and 0.350 on Large-eight, while DDPG+HER gives 0.290 on Large-all and 0.280 on Large-eight.
  • Cap, Q3. Cap 2 is one shared default, chosen for the highest cross-task mean in Table 2. Caps of 0, 2, and 20 give 0.360, 0.600, and 0.668 on obstacle Pick-and-Place and 0.908, 0.928, and 0.800 on CustomLarge: obstacles reward a stronger check and long mazes a moderate one, consistent with the advantage bound, as discussed in the response to ifwN.

Thank you for the detailed response, the additional baselines on the Large layouts and the TD4 + hinge control.

Before I reassess, could you please provide a revised PDF with the changes highlighted in color? Several of the promised changes concern the presentation (terminology in Sec. 4.1, actor gradient and Algorithm 1 in Sec. 4.3, revised claims in Sec. 5.1), and I would like to check them in the manuscript itself.

Regarding the significance test, could you clarify what exactly the paired bootstrap resamples (seeds, evaluation episodes or tasks)? With 3 to 5 seeds per task, the reported confidence intervals seem narrow. Please also state whether all tasks use the same runs, since Table G1 seems to combines a new batch with the submitted batch.

Thank you. The revised PDF is uploaded with changes highlighted in color purple for your review, blue for shared revisions; the items you list are in Sec. 4.1, Sec. 4.3 and Sec. 5.1.

The bootstrap resamples paired training runs, not episodes or tasks. On each task, each CGCPO run is paired with the TD4 and TD4 + hinge runs of the same seed, giving five pairs per method and task. Each of 50,000 replicates resamples these pairs within each task and averages the ten task means with equal weight. The interval is narrow because it covers this ten-task average. Per-task intervals in Table 3 are wider. All tasks now use the same runs. The revision replaces Table G1 with a single batch, the additional batch in Tables 6 and 7, and Appendix A.1 and Table 3 give the protocol and the updated intervals.

Reviewer_ifwNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces an RL algorithm that uses an action chunking flow-matching imitation policy to provide long-range estimates of the current policy’s behavior. It is interesting because the primitive policy and the flow matching “action generator” maintain fully separate parameters, but critics trained on each are used to update the primitive behavior policy. The authors provide a theoretical derivation showing that the optimality gap of the primitive policy is bounded by a quantity that goes down when the critic trained on the action chunking flow-matching policy gives near-optimal scores to the actions that would be chosen by a reference primitive policy (and other terms, but that is the part of their theorem that I find most interesting). The empirical evaluation shows that their approach ties the best baseline (EBP) in the Fetch manipulation tasks, and beats baselines in maze navigation tasks.

Critical strengths
  1. The paper relates action chunking flow models, which are commonly used in imitation learning, to provide additional supervision to a primitive RL policy. This seems novel, and is an interesting way to take advantage of two different policy architectures’ strengths.
  2. There is a theoretical justification for why the critic trained on the action-chunking imitation policy is useful for training the primitive RL policy.
  3. The empirical evaluation shows performance on both manipulation tasks and maze navigation tasks.
Critical weaknesses
  1. The paper’s technical content (algorithm implementation and mathematical proofs) is not presented clearly and is hard to understand. For example, there is no text that provides detailed comparison and contrasting to the closest existing RL algorithms and proofs. Section 4.3 is titled “Constrained Policy Optimization” but there is no explicit comparison to the commonly known RL algorithm of the same name. There is no pseudocode for the RL algorithm. This makes it hard for me to understand if the math is correct and relevant.
  2. The paper’s nontechnical content is also presented in a disorganized fashion: for example the Related Work section is one very long paragraph with lots of individual sentences summarizing what other approaches do. There are some small component-level comparisons to CGCPO, but not enough to give me a grounded sense of where this approach fits in the literature.
Questions for authors
  1. What about the algorithm is specific to Goal-Conditioned RL? I see of course that it is a GCRL algorithm, but it seems like the core idea of using a flow-matching policy and associated critic could also be evaluated in vanilla RL. Would your approach also apply to vanilla RL? Is hindsight relabeling crucial to run into the situation where the flow matching critic is actually more useful than just using a primitive critic on its own?
  2. What existing proofs in the literature are most similar to the proofs in your theoretical derivation? Can you isolate out a small piece that is novel, which would make it easier for readers to understand? The primary reason for my low score is that I found the presentation of the paper difficult to understand, it took me a very long time to digest the proofs and I am still not certain I have checked them fully.
  3. What is the value of showing Proposition 1 (that the value functions converge to their fixed points under fixed policies and fixed data) when in practice updates are happening to the policies (and in fact the values of the sequence critic are used to update the primitive policy, introducing non-stationarity). Are proofs of similar theorems common in RL literature, or is this a weaker statement than is typical?
  4. Is it true that the generator policy is essentially trained with imitation learning over the replay buffer? Under what circumstances are the rewards from this policy better than what you would get just from typical Bellman credit assignments from a primitive behavior policy? I see you have some improvement in the maze evaluations, but I am not sure exactly what circumstances I would expect this technique to help.
  5. (Extra experiments): Are there any manipulation tasks where your approach would beat all your baselines?
2 comments

We thank Reviewer ifwN for recognizing the novelty of supervising a primitive policy with action-chunk models.

1. Algorithm, CPO, and related work (W1, W2)

Our algorithm turns the long-term value of imagined action sequences into a gradient for the current action and checks that action with a one-step critic.

Actor gradient (now in Sec. 4.3). With the reference action a−=π−(s,g)a^-=\pi^-(s,g) fixed,

∇θL=E(s,g)[(−∇aq^H(s,g,a)−λI[Δ^1<τ]∇aQϕ(s,g,a)+∇a∥a∥22)∣a=πθ(s,g)∇θπθ(s,g)],\nabla_\theta\mathcal{L}=\mathbb{E}_{(s,g)}\Big\lbrack \big(-\nabla_a\hat q_H(s,g,a)-\lambda\mathbb{I}\lbrack \hat\Delta_1\lt \tau\rbrack \nabla_aQ_\phi(s,g,a)+\nabla_a\lVert a\rVert_2^2\big)\big\vert_{a=\pi_\theta(s,g)}\nabla_\theta\pi_\theta(s,g)\Big\rbrack ,

where q^H(s,g,a)=EϵQH(s,g,Aω(s,g,a,ϵ))\hat q_H(s,g,a)=\mathbb{E}_{\epsilon}Q^H(s,g,A_\omega(s,g,a,\epsilon)) backpropagates through the frozen generator and Δ^1=Qϕ(s,g,πθ)−Qϕ(s,g,a−)\hat\Delta_1=Q_\phi(s,g,\pi_\theta)-Q_\phi(s,g,a^-).

Algorithm 1 (one training cycle).

  1. Collect episodes with πθ\pi_\theta plus DDPG exploration noise.
  2. Sample complete HH-step blocks; relabel goals with HER (ratio 4/5).
  3. Fit the flow-matching generator AωA_\omega, the sequence critic QHQ^H (Eq. 1) and the one-step critic QϕQ_\phi (Eq. 2) with target actor π−\pi^-.
  4. Freeze them and update θ\theta with the gradient above.
  5. Set λ←min⁡(2,λ+0.02E[(τ−Δ^1)+])\lambda\leftarrow\min(2,\lambda+0.02\mathbb{E}\lbrack (\tau-\hat\Delta_1)_+\rbrack ) with τ=0\tau=0; update EMA targets (polyak 0.95).
  • CPO. CPO bounds a cumulative auxiliary cost through trust-region updates. Our constraint Δ^1≥τ\hat\Delta_1\ge\tau has no auxiliary cost: it compares the task returns of the candidate and the target actor's action under the same reference policy, a per-state monotone-improvement check in the spirit of Kakade & Langford (2002), enforced by a capped hinge. It is needed because generated continuations can differ from what the feedback policy executes.
  • Related work. Sec. 2 now has four paragraphs (relabeling and multi-step credit, action-chunk critics, generative proposals, reference-constrained updates), contrasting where each method combines short and long horizons.

2. Theory Q2, Q3

Our theory asks how good the current action is when imagined long-term value and the one-step check meet in one score.

  • Standard parts. Proposition 1 is the standard per-policy Bellman contraction and defines what imagined-block value estimates; the Bellman-error bounds and the comparison turning an action gap into a value gap follow Kakade & Langford (2002), and with π+=π−\pi^+=\pi^- Theorem 1 reduces to the classical residual bound.
  • What it reveals. The block operator leaves a discrepancy DHD_H from the reference value, so even with zero fitting error the two critics evaluate different continuations (eS=(ϵH+DH)/(1−γH)e_S=(\epsilon_H+D_H)/(1-\gamma^H)); this is why the current action needs a one-step check.
  • The new step (Eqs. C.23–C.26). With d(x)d(x) the true reference-value gap of the chosen action, c=η+2eSc=\eta+2e_S and e1e_1 the one-step error bound, d≤c+λmax⁡(2e1−d,0)d\le c+\lambda\max(2e_1-d,0). The same ingredients give Adv−(x)≥−(c+2λe1)/(1+λ)\mathrm{Adv}^-(x)\ge-(c+2\lambda e_1)/(1+\lambda), which tightens with λ\lambda whenever c>2e1c\gt 2e_1; Sec. 4.4 states it after a proof roadmap.

3. Q1, Q4, Q5

Our method targets sparse-reward obstacle tasks, where actions with equal immediate reward lead to different futures, and lets imagined long-term value tell them apart for the current action.

  • Q1. The objective needs neither goals nor HER (x=sx=s gives vanilla RL); HER belongs to the sparse-reward setting we target and supplies the successful continuations the generator imitates.
  • Q4. Yes, the generator is behavior cloning of four-action replay blocks conditioned on (s,g,a0)(s,g,a_0), with no dynamics model. It beats one-step credit when actions share a first reward but not their futures: one-step backups propagate that difference step by step, while the block value exposes it at once. CustomLarge success climbs from 0.476 (one-step DDPG+HER) to 0.916, and at the same hinge imagined guidance adds 0.029 [0.015, 0.043] over TD4 (Table G1).
  • Q5. Yes, obstacle Push: in submitted Fig. 3 our method has the highest mean (0.819; EBP 0.817; DDPG+HER and TMD 0.280); EBP prioritizes replay, orthogonal to our actor update. On obstacle Pick-and-Place our method reaches 0.660 against 0.620 (TD4 + hinge), 0.360 (DQC) and 0.340 (TD4).
Reviewer_ifwNOpenReview ↗

Thanks for clarification, revised manuscript still needs better presentation

I would like to thank the authors for their responses to my questions. My questions 2, 3, and 4 are resolved, and I appreciate the authors adding comparison to existing proofs in the literature.

The author’s response to my question 1 seems to confirm my observation that the core idea of the paper is not tightly related to the GCRL setting. The author’s response is very short, and does not shed light on why this approach is presented as a GCRL algorithm, when the core idea could be applied to GCRL. It would be good for the paper to clearly pick apart what parts of the approach are uniquely useful for GCRL. Also, in the authors’ response they say the objective does not need HER. This is fine, but I would have liked the authors to explain more, so I would understand why they include hindsight relabeling in their algorithm (step 3 of Algorithm 1 in the revised manuscript).

Overall, this paper is not clearly organized. This is shown by the unclear purpose of the core techniques (would they transfer to vanilla RL, what parts of the GCRL setup are crucial for this approach). The presentation is also unclear throughout. For example, during the early sections of the paper, concrete hyperparameters (“the target actor keeps 0.95…”, line 145), eval environments (“Fetch and PointMaze are deterministic…”, line 296, which appears during the “Theoretical Analysis” section) are mentioned, without much introduction. While these details are good to include in the paper, including them in the wrong sections makes the paper hard to follow. If the theory depends on some assumptions on model dynamics, those should be stated clearly in the problem setup- concrete Gym environments should not be mentioned in the theory section (unless it is part of a running illustrative example, which this is not).

The algorithm pseudocode is also not formatted nicely and leaves some questions open. For example, the role of the blocks of length H. The notion of a block of length H is first introduced in line 143 and makes sense at a very high level, but it is not clear whether these blocks are tumbling windows, sliding windows, or full episodes, or some other structure. Line 149 talks about blocks of length H as an imagined sequence, which makes sense from an inference perspective, but does not resolve the ambiguity for the training. The algorithm pseudocode should resolve this ambiguity, but it does not.

Also, terminology and symbols seem like they might not always be consistent. See section 4.1 which introduces the “action chunk critic” but in the rest of the paper it seems to be called “sequence critic”. Also, some algorithmic techniques are not clearly introduced: in section 4 there is on-the-fly introduction of the concept of “target critics” etc., which is a fine and normal algorithmic technique, but should be signposted earlier in the paper.

Due to these (and other similar) presentational issues I maintain my score of 2. I believe the technical content of the paper is interesting and could make a good paper if the contributions were contextualized better and the presentation were clearer.

Comments on the paper

1 comment

We have uploaded a revised manuscript with all changes highlighted in color, blue marks revisions shared across reviews, and purple, teal and green mark changes for Reviewers e2Cg, ifwN and xLfD, respectively. Main updates: standard terminology and the definition of imagined sequences (Sec. 4.1), the actor gradient and Algorithm 1 (Sec. 4.3), a restructured Related Work (Sec. 2), task by task comparisons including TD4 and TD4 + hinge (Sec. 5.1), TD4 in the coverage analysis (Sec. 5.3), and horizon and hinge cap ablations (Sec. 5.4). We thank all reviewers for their feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.