- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This article proposes the Supportive Proportion Guidance (SPG) method, which treats the guidance intensity and sample allocation as a joint decision within each mini-batch to better utilize the planning regularization strategy for updating, in order to accelerate the goal-conditioned policy learning.
Critical strengths
The main advantage of this article is that the writing is clear and easy to understand, and it explores the application of CONSTRAINED OPTIMIZATION AND ALLOCATION in the acceleration of goal-conditioned RL based on subgoals.
Critical weaknesses
My main concern is that I haven't grasped the significance of the acceleration strategy learning mentioned in this article and the innovative points of the article either.
-
Although Figure 1 illustrates the problems of the previous methods and the advantages and methods of the SPG proposed in this paper, the authors did not conduct any tests in actual tasks, so they are not aware of the significance of the mentioned problems.
-
Can I understand how the method described in this article handles the trade-off between RL and planning in terms of learning and adjustment? Or, how do we learn the parameters? I don't know where the significance of doing this lies. If it's merely to accelerate learning, I think it might not be that important in goal-conditioned RL.
-
From the experimental results shown in Figure 3, it seems that the proposed SPG merely learns faster, but the final success rate has not improved compared to the previous methods. Therefore, I do not consider the contribution of this paper to be significant.
-
The experimental environments are rather limited, only including antmaze and fetch series. So they are a bit monotonous. Extending to more environments is definitely necessary.
-
This article lacks more visualizations and experiments to demonstrate where their method works and the significance of the research allocation.
Questions for authors
- Could the author conduct more experiments in different environments to test the validity of the method, such as JaxGCRL?
JaxGCRL: Accelerating Goal-Conditioned Reinforcement Learning Algorithms and Research. ICLR, 2025.
- Could the author add the following most relevant baselines to demonstrate the effectiveness of their method? For example: [1, 2, 3]
[1] Breadth-First Exploration on Adaptive Grid for Reinforcement Learning. ICML, 2024.
[2] Strict subgoal execution: Reliable long-horizon planning in hierarchical reinforcement learning. ICLR, 2026.
[3] Novelty-aware graph traversal and expansion for hierarchical reinforcement learning. CIKM, 2024.
-
I think SPG is more of a plug-and-play approach rather than a new subgoal-based strategy learning method. Additionally, I'm not sure how this method would perform when applied to previous methods. Could the author please explain this?
-
How did SPG perform in the random goal environment? For example, in the configuration of BEAG?
BEAG: Breadth-First Exploration on Adaptive Grid for Reinforcement Learning. ICML, 2024.
- The relevant parameters of SPG are four in number. This not only increases the complexity but also obscures the contribution of the algorithm, making it difficult to explain and challenging to optimize. Has the author considered this issue?
2 comments
Mechanism, parameters, planning sources, and random-goal evaluation
4. How SPG adjusts planning guidance. At each replay update, SPG evaluates K planning-induced candidate actions against the current policy action. The number of positive comparisons is S_i. With m=ceil(eta K), the number of contexts satisfying S_i>=m determines the total guidance coefficient C. SPG then solves sum_i min(1,lambda S_i)=C over positive-support contexts. The actor uses complementary coefficients q_i and 1-q_i for planning guidance and return learning, while retaining the complete planning mixture.
Thus the current batch determines both the overall guidance level C/B and the distribution of that guidance. The total responds to the prevalence of sufficient support, while the shares respond to its relative strength. Theorem 1 makes this distinction precise through and bounded influence of one changed candidate comparison. These are properties of the allocation mechanism, not additional networks to train.
5. The parameters have distinct and directly interpretable roles. In the controlled implementation:
| Quantity | Role | Setting |
|---|---|---|
| K | Planning candidates per context | 16 |
| B | Replay minibatch size | 2048 |
| eta | Support fraction defining m | 0.25 |
| alpha | KL-loss scale | 0.1 |
The threshold m=4 is derived from eta and K; lambda is solved from the current total constraint rather than tuned or learned. SPG adds no weighting network or separate weighting optimizer. Figure 5's AntMaze-U sweep shows why guidance strength matters:
| Guidance scale kappa | 0 | 1 (default) | 1.6 |
|---|---|---|---|
| Reported success | 10.3% | 84.0% | 62.7% |
More planning supervision does not automatically improve learning because its complementary coefficient also changes the return contribution.
6. Existing diagnostics show where the allocation acts. Table 4 applies SPG to 20 stored control-policy batches, containing 40,960 contexts:
| Support count S_i | Mean SPG coefficient |
|---|---|
| 0 | 0.0000 |
| 1 | 0.1421 |
| 2 | 0.2843 |
| 4 | 0.5689 |
| 6 | 0.8535 |
| 8–16 | 1.0000 |
This profile shows graded allocation rather than a binary accept/reject decision: zero support receives no guidance, weak support receives limited guidance, and stronger support can saturate the cap. Figure 7 separately reports guidance totals and eligible fractions along actual SPG training trajectories. Together with the task-level controls in our first response, these diagnostics connect the allocation rule to observable coefficients and learning outcomes.
7. Integration across planning sources is part of the contribution. We study how a learner uses a planning reference, so compatibility with existing planning sources is an intended benefit. Figure 3 uses learned RIS references; Figs. 4(a)–(b) retain the graph-planning procedure and compare PIG with SPG-PIG. The latter pair improves success on both reported navigation layouts. These two sources test the learning interface with different reference constructions. The SPG allocator operates during training and adds no SPG allocation module to deployment; this does not require changing the underlying method's execution protocol.
8. Random goals and broader evaluation. Figure 3 already evaluates random start–goal pairs across six navigation and manipulation tasks. Figure 5 uses fixed goals for allocation controls, whereas Appendix C.2 separately evaluates random and farthest goals. The five-seed C.2 random-goal component reaches:
| SPG random-goal terminal | AntU | AntFg |
|---|---|---|
| Mean success | 100.0% | 99.4% |
Its learning-efficiency gains are reported in our first response.
We appreciate the suggestions concerning JaxGCRL, BEAG, strict subgoal execution, and novelty-aware graph traversal. We plan to broaden environment coverage in the camera-ready version, using JaxGCRL as a benchmark infrastructure. BEAG, SSE and NGTE address graph construction, exploration or subgoal execution. Our present RIS and PIG comparisons isolate the benefit of changing how a given planning reference supervises policy learning. This complements work on planning, exploration, and execution, and gives a concrete basis for extending SPG to further systems.
Task-level evidence for guidance allocation and learning efficiency
Thank you for raising the significance of the allocation problem. We would like to clarify where the paper tests this question in actual learning tasks and provide the controlled results in a more explicit form.
1. The allocation problem is tested beyond the schematic in Fig. 1. Figure 5(c) compares allocation rules during policy learning on a fixed-goal navigation task. All three methods share the same full-mixture reference construction and complementary actor objective. The paper reports the following success rates over three training seeds:
Shuffled preserves each observed batch's coefficient multiset and total while changing which eligible contexts receive the coefficients. The 27.7-percentage-point difference from Uniform, together with the Shuffled comparison, provides task-level evidence that assigning guidance according to local support matters.
The separate Appendix C.2 study comprises 40 runs: two layouts, four allocation rules, and five seeds. Every rule uses the full planning mixture and the same complementary loss. UE distributes the support-derived total uniformly over positive-support contexts; UB distributes it over the entire batch; FE assigns a fixed coefficient to eligible contexts. SPG uses capped support-proportional shares. Rules are recomputed on their respective training batches.
Values are means ± sample SD over five training seeds. Joint success is min(random, farthest) at each checkpoint, integrated over 50K–150K on AntU and 50K–400K on AntFg. The layout-equal quantity first averages the two layouts within each seed. AntU and AntFg are the archived control-study names.
SPG has the highest layout-averaged AUC: 13.1% above UE and 21.6% above FE. Its paired AUC difference from UB is positive in all five seed blocks on each layout. These common-objective comparisons directly demonstrate that guidance allocation has learning consequences; the effects cannot be explained solely by changing the loss formulation.
2. Learning efficiency is a measurable policy-learning benefit. In sparse-reward tasks, learning a useful policy with fewer environment interactions reduces the experience required to acquire that behavior. The random-goal component of the five-seed control study gives:
All five seeds reach the threshold. On AntFg, SPG reaches it earlier in four paired seeds and at the same checkpoint in the fifth. The threshold is the first observed crossing on the 5K evaluation grid. This decomposition supplements the joint-AUC results above.
3. The main results also include terminal gains. Figure 3(f) shows a terminal success gap over RIS and SPReD on FetchPickAndPlace, while Figs. 4(a)–(b) show higher success for SPG-PIG than PIG at the reported budgets. On tasks where guided methods eventually saturate, AUC and interactions to a success threshold capture the remaining practical difference. The contribution therefore concerns how effectively a policy learns from an available planning reference within its interaction budget.