- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This article proposes a method called Continuation Guided Constrained Policy Optimization (CGCPO) based on the fact that the current GCRL method cannot effectively distinguish between useful and blocked actions at obstacles. It combines imagined sequence guidance with a separate comparison of current action values. CGCPO adds a generator on ddpg+her to generate actions quickly, a sequence value to score the quickness of actions, and an action loss to improve the difference in sequence values of generated blocks under candidate and reference actions, and a penalty is applied when the one-step critic value is below the boundary .
Critical strengths
The initial insight of this article is excellent. It identified the issue that sparse rewards in the presence of obstacles result in the same rewards for different actions, and proposed a constraint strategy optimization based on continuation guidance to learn useful actions while avoiding blocking actions. The writing is clear and coherent, the diagrams are easy to understand, and it thoroughly discusses the connections with existing methods. Moreover, it has achieved advanced results in the fetch and pointmaze environments.
Critical weaknesses
-
Although the author's proposed method is reasonable and logical, the idea of generating mirror sequences/tracks and the values of the sequences is indeed already an existing work in RL and has been well studied. Therefore, I do not consider the method to be particularly novel. Moreover, the second term of the penalty for the action loss (Equation 5) for the one-step critic is also similar to the previous CQL idea.
-
I wonder how the sequence generator can ensure the correctness of the sequence without using a dynamics model for learning? Moreover, the sequence branch (generator + sequence critic) is conceptually close to the n-step TD error, but the key contrast between the two is not provided by the authors. It may have real advantages, such as using an entire block as input to make the variance of the regression target smaller, but this requires experimental proof.
For example, the data in the paper:
1). In Table 1, on the obstacle Pick&Place, the last-five of pure sequence guidance (NoPenalty) is 0.360±0.022, which is exactly the same as TD4's 0.360±0.048. On CustomLarge, the two are 0.908 vs 0.864, with a difference of approximately 1.7 standard errors.
2). In Table 2, by replacing flow with CGCPO_Det, which is deterministic mean block regression, the average of six tasks is 0.843, which is almost the same as Cap20's 0.847. Generative modeling is not necessary.
3). In FullGrad, where the first action is treated as a policy action (this is exactly how the n-step critic handles the first action), it achieves the highest score of 0.776 on the obstacle Pick&Place, which is higher than the default generative score of Cap20 (0.668).
-
The hyperparameters in Appendix A.1 are almost exactly the same as the default configuration of HER in the OpenAI Gym baselines:
polyak 0.95,noise 0.2,random actions 0.3,re-labeling 4/5,40 updates per cycle,observation clipping ±5,target clipping [−1/(1−γ), 0]. However, this configuration's actor loss has a default action quadratic penalty ofaction_l2=1.0, which is not included in Equation (5). If the baseline retains this term, the coverage gap may largely come from "actions being compressed by L2", rather than credit assignment. The authors need to clarify this. -
Theorem 1 has two issues.
First, it fails to explain the role of punishment. From , it can be seen that when is fixed, gives the tightest bound. The punishment only adds terms and does not subtract them. However, in the experiment, the punishment is precisely the most crucial component (removing it causes 5 out of 6 tasks to decline).
Second, it cannot distinguish CGCPO from TD4. The advantage of sequential branching is only reflected in being less than , which holds for any -step critic.
-
The fairness and coverage of the experiments.
1). Antmaze only has the "large" series of tasks. I suggest adding more maze experiments to demonstrate the performance and contribution of CGCPO, such as JaxGCRL [1].
[1] Accelerating goal-conditioned reinforcement learning algorithms and research, ICLR, 2025.
2). The most relevant action chunk value methods (Q-chunking, DQC, Chunk-Guided Q-Learning) are only distinguished as "offline, single critic" in related works, and no experimental comparisons have been conducted. The author has transformed CQN-AS into an online HER version, indicating that offline/online status does not constitute a reason for incomparable results.
This sole sequence baseline is almost 0 on Push as seen in Figure 3(b), and the evaluation protocol is inconsistent in the two domains (Appendix A.1), making it difficult to be regarded as a strong baseline.
3). The most straightforward n-step DDPG+HER only appears in ablation studies and has not entered the main comparison.
4). H has never conducted ablation studies and has not reported the computational cost.
Questions for authors
- "TD4 critic + the same step penalty" (without using the generator) - how effective is this? If it is comparable to CGCPO, what are the irreplaceable advantages of the generator?
- Is the environment transfer random? If it is deterministic, executing the same action block in both open-loop and closed-loop will result in the same trajectory. "Open-loop mismatch" is actually a deviation from the strategy in terms of deviation from the strategy. If it is random, using the closed-loop recorded blocks to regress to the open-loop value will have selection bias, which is inconsistent with the setting of Proposition 1. How to handle this?
- Do DDPG + HER, CGCPO_Prim, and CGCPO include action L2 regularization? After removing it, how is the reachability of DDPG + HER on CustomLarge?
- How to handle the transfer at the end of the round that is less than H steps? If these starting points are excluded, does the step that truly reaches the end from the original goal never enter the training of the one-step critic?
- In Figure 4(a), is approximately 1e-3, while in Figure 11(d) on the same task it is around 1. What are the reference strategies at these two locations? How is the obstacle version of the Fetch task defined?
We thank Reviewer xLfD for calling our starting insight excellent: near obstacles, sparse rewards give useful and blocked actions the same reward, and imagined long-term value tells them apart for the current action.
1. Novelty and the source of the gain: W1, W2, Q1
Our contribution is where sequence values act: imagined long-term value guides a one-step policy's current action, kept sound by a one-step check.
Table R2. Obstacle Pick-and-Place, last-five success, 5 seeds.
2. Theory and dynamics: W4, Q2
Our theory certifies the current action chosen by imagined long-term value under the one-step check.
3. Implementation details: W3, Q3–Q5
Our gains come from crediting the current action with imagined long-term value, on a backbone all methods share.
Table R3. DDPG+HER on CustomLarge, 5 seeds.
4. Baselines, horizon, and compute: W5
Our method draws long-term credit from imagined action sequences, so we vary the horizon and add chunk-value baselines.
Table R4. Horizon sweep and DQC, last-five success, 5 seeds.