acceptodds
Under review as a conference paper at ICLR 2027

Local Evidence, Joint Allocation: Planning Guidance for Goal-Conditioned Reinforcement Learning

Abstract

Planning can accelerate sparse-reward goal-conditioned reinforcement learning by using subgoal-conditioned behavior to regularize policy updates. Yet assigning guidance weights independently across samples couples their relative priorities with the overall balance between planning supervision and return maximization. We propose Support-Proportional Guidance (SPG), which treats guidance strength and sample allocation as a joint decision within each minibatch. A critic compares planning-induced actions with the current policy's sampled action to obtain a support count for each state-goal pair. The prevalence of sufficient support sets the mean guidance weight, while capped proportional allocation distributes the corresponding total according to support strength. The actor retains the complete planning-induced action mixture as its reference, so critic evidence changes guidance strength rather than selecting individual target actions. We establish that weak local support limits an individual guidance weight and that one changed candidate comparison has bounded influence on the entire allocation. Across six navigation and manipulation tasks, SPG improves sample efficiency over the evaluated baselines. Further results show that both excessively weak and excessively strong guidance reduce the success rate and that SPG outperforms general sample allocation methods.

Primary area:
reinforcement learning
Keywords:
Goal-Conditioned Reinforcement Learning, Planning Guidance, Policy Regularization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FswUOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This article proposes the Supportive Proportion Guidance (SPG) method, which treats the guidance intensity and sample allocation as a joint decision within each mini-batch to better utilize the planning regularization strategy for updating, in order to accelerate the goal-conditioned policy learning.

Critical strengths

The main advantage of this article is that the writing is clear and easy to understand, and it explores the application of CONSTRAINED OPTIMIZATION AND ALLOCATION in the acceleration of goal-conditioned RL based on subgoals.

Critical weaknesses

My main concern is that I haven't grasped the significance of the acceleration strategy learning mentioned in this article and the innovative points of the article either.

  1. Although Figure 1 illustrates the problems of the previous methods and the advantages and methods of the SPG proposed in this paper, the authors did not conduct any tests in actual tasks, so they are not aware of the significance of the mentioned problems.

  2. Can I understand how the method described in this article handles the trade-off between RL and planning in terms of learning and adjustment? Or, how do we learn the parameters? I don't know where the significance of doing this lies. If it's merely to accelerate learning, I think it might not be that important in goal-conditioned RL.

  3. From the experimental results shown in Figure 3, it seems that the proposed SPG merely learns faster, but the final success rate has not improved compared to the previous methods. Therefore, I do not consider the contribution of this paper to be significant.

  4. The experimental environments are rather limited, only including antmaze and fetch series. So they are a bit monotonous. Extending to more environments is definitely necessary.

  5. This article lacks more visualizations and experiments to demonstrate where their method works and the significance of the research allocation.

Questions for authors
  1. Could the author conduct more experiments in different environments to test the validity of the method, such as JaxGCRL?

JaxGCRL: Accelerating Goal-Conditioned Reinforcement Learning Algorithms and Research. ICLR, 2025.

  1. Could the author add the following most relevant baselines to demonstrate the effectiveness of their method? For example: [1, 2, 3]

[1] Breadth-First Exploration on Adaptive Grid for Reinforcement Learning. ICML, 2024.

[2] Strict subgoal execution: Reliable long-horizon planning in hierarchical reinforcement learning. ICLR, 2026.

[3] Novelty-aware graph traversal and expansion for hierarchical reinforcement learning. CIKM, 2024.

  1. I think SPG is more of a plug-and-play approach rather than a new subgoal-based strategy learning method. Additionally, I'm not sure how this method would perform when applied to previous methods. Could the author please explain this?

  2. How did SPG perform in the random goal environment? For example, in the configuration of BEAG?

BEAG: Breadth-First Exploration on Adaptive Grid for Reinforcement Learning. ICML, 2024.

  1. The relevant parameters of SPG are four in number. This not only increases the complexity but also obscures the contribution of the algorithm, making it difficult to explain and challenging to optimize. Has the author considered this issue?
2 comments

Task-level evidence for guidance allocation and learning efficiency

Thank you for raising the significance of the allocation problem. We would like to clarify where the paper tests this question in actual learning tasks and provide the controlled results in a more explicit form.

1. The allocation problem is tested beyond the schematic in Fig. 1. Figure 5(c) compares allocation rules during policy learning on a fixed-goal navigation task. All three methods share the same full-mixture reference construction and complementary actor objective. The paper reports the following success rates over three training seeds:

AllocationReported success
Uniform56.3%
Shuffled38.0%
SPG84.0%

Shuffled preserves each observed batch's coefficient multiset and total while changing which eligible contexts receive the coefficients. The 27.7-percentage-point difference from Uniform, together with the Shuffled comparison, provides task-level evidence that assigning guidance according to local support matters.

The separate Appendix C.2 study comprises 40 runs: two layouts, four allocation rules, and five seeds. Every rule uses the full planning mixture and the same complementary loss. UE distributes the support-derived total uniformly over positive-support contexts; UB distributes it over the entire batch; FE assigns a fixed coefficient to eligible contexts. SPG uses capped support-proportional shares. Rules are recomputed on their respective training batches.

AllocationAntU joint AUCAntFg joint AUCLayout-equal AUC
SPG0.4286 ± 0.15130.2636 ± 0.08510.3461 ± 0.1006
UE0.3698 ± 0.05580.2424 ± 0.08090.3061 ± 0.0618
FE0.4290 ± 0.12730.1403 ± 0.16790.2846 ± 0.1222
UB0.1048 ± 0.07360.0464 ± 0.10330.0756 ± 0.0729

Values are means ± sample SD over five training seeds. Joint success is min(random, farthest) at each checkpoint, integrated over 50K–150K on AntU and 50K–400K on AntFg. The layout-equal quantity first averages the two layouts within each seed. AntU and AntFg are the archived control-study names.

SPG has the highest layout-averaged AUC: 13.1% above UE and 21.6% above FE. Its paired AUC difference from UB is positive in all five seed blocks on each layout. These common-objective comparisons directly demonstrate that guidance allocation has learning consequences; the effects cannot be explained solely by changing the loss formulation.

2. Learning efficiency is a measurable policy-learning benefit. In sparse-reward tasks, learning a useful policy with fewer environment interactions reduces the experience required to acquire that behavior. The random-goal component of the five-seed control study gives:

LayoutSPG / UE random AUCSPG / UE interactions to 80%Interaction reduction
AntU0.8311 / 0.796896K / 103K6.8%
AntFg0.8452 / 0.8030155K / 188K17.6%

All five seeds reach the threshold. On AntFg, SPG reaches it earlier in four paired seeds and at the same checkpoint in the fifth. The threshold is the first observed crossing on the 5K evaluation grid. This decomposition supplements the joint-AUC results above.

3. The main results also include terminal gains. Figure 3(f) shows a terminal success gap over RIS and SPReD on FetchPickAndPlace, while Figs. 4(a)–(b) show higher success for SPG-PIG than PIG at the reported budgets. On tasks where guided methods eventually saturate, AUC and interactions to a success threshold capture the remaining practical difference. The contribution therefore concerns how effectively a policy learns from an available planning reference within its interaction budget.

Mechanism, parameters, planning sources, and random-goal evaluation

4. How SPG adjusts planning guidance. At each replay update, SPG evaluates K planning-induced candidate actions against the current policy action. The number of positive comparisons is S_i. With m=ceil(eta K), the number of contexts satisfying S_i>=m determines the total guidance coefficient C. SPG then solves sum_i min(1,lambda S_i)=C over positive-support contexts. The actor uses complementary coefficients q_i and 1-q_i for planning guidance and return learning, while retaining the complete planning mixture.

Thus the current batch determines both the overall guidance level C/B and the distribution of that guidance. The total responds to the prevalence of sufficient support, while the shares respond to its relative strength. Theorem 1 makes this distinction precise through qi≤min⁡(1,Si/m)q_i\leq\min(1,S_i/m) and bounded influence of one changed candidate comparison. These are properties of the allocation mechanism, not additional networks to train.

5. The parameters have distinct and directly interpretable roles. In the controlled implementation:

QuantityRoleSetting
KPlanning candidates per context16
BReplay minibatch size2048
etaSupport fraction defining m0.25
alphaKL-loss scale0.1

The threshold m=4 is derived from eta and K; lambda is solved from the current total constraint rather than tuned or learned. SPG adds no weighting network or separate weighting optimizer. Figure 5's AntMaze-U sweep shows why guidance strength matters:

Guidance scale kappa01 (default)1.6
Reported success10.3%84.0%62.7%

More planning supervision does not automatically improve learning because its complementary coefficient also changes the return contribution.

6. Existing diagnostics show where the allocation acts. Table 4 applies SPG to 20 stored control-policy batches, containing 40,960 contexts:

Support count S_iMean SPG coefficient
00.0000
10.1421
20.2843
40.5689
60.8535
8–161.0000

This profile shows graded allocation rather than a binary accept/reject decision: zero support receives no guidance, weak support receives limited guidance, and stronger support can saturate the cap. Figure 7 separately reports guidance totals and eligible fractions along actual SPG training trajectories. Together with the task-level controls in our first response, these diagnostics connect the allocation rule to observable coefficients and learning outcomes.

7. Integration across planning sources is part of the contribution. We study how a learner uses a planning reference, so compatibility with existing planning sources is an intended benefit. Figure 3 uses learned RIS references; Figs. 4(a)–(b) retain the graph-planning procedure and compare PIG with SPG-PIG. The latter pair improves success on both reported navigation layouts. These two sources test the learning interface with different reference constructions. The SPG allocator operates during training and adds no SPG allocation module to deployment; this does not require changing the underlying method's execution protocol.

8. Random goals and broader evaluation. Figure 3 already evaluates random start–goal pairs across six navigation and manipulation tasks. Figure 5 uses fixed goals for allocation controls, whereas Appendix C.2 separately evaluates random and farthest goals. The five-seed C.2 random-goal component reaches:

SPG random-goal terminalAntUAntFg
Mean success100.0%99.4%

Its learning-efficiency gains are reported in our first response.

We appreciate the suggestions concerning JaxGCRL, BEAG, strict subgoal execution, and novelty-aware graph traversal. We plan to broaden environment coverage in the camera-ready version, using JaxGCRL as a benchmark infrastructure. BEAG, SSE and NGTE address graph construction, exploration or subgoal execution. Our present RIS and PIG comparisons isolate the benefit of changing how a given planning reference supervises policy learning. This complements work on planning, exploration, and execution, and gives a concrete basis for extending SPG to further systems.

Reviewer_AY65OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces how to integrate planning-derived references into goal-conditioned reinforcement learning by coordinating the per-sample guidance coefficients used in policy regularization. It introduces Support-Proportional Guidance (SPG), which uses a target twin-critic to count how many planning-induced actions beat the current policy’s sampled action. The number of well-supported contexts sets the total planning supervision per mini batch, and capped proportional fairness allocates it across samples while keeping the full planning mixture as the KL reference. It also proves support-dependent weight envelopes and bounded influence of a single comparison, and shows an equivalence to a KL projection. The experimental evaluation spans six navigation and manipulation tasks and two planning sources (RIS, PIG), and shows better sample efficiency than fixed regularization (RIS), uncertainty-weighted regularization (SPReD), and PCGrad, plus controls on guidance strength and allocation

Critical strengths

• The paper introduces a clear and principled batch-coupled allocation mech-anism that decouples relative per-sample priorities from the total amount of planning supervision which addresses a key issue of independently set sample weights, while retaining the full planning-induced action mixture as the KL reference, aligning the supervision target with the planner’s structured uncertainty.

• This allocation mechanism is characterized both as a capped proportional-fairness optimizer and as an information projection under constraints.

Critical weaknesses

• The total guidance is derived using a threshold on binary support counts which is heuristic and ignores value gaps and uncertainty. The theoretical gain over uniform allocation holds only if support correlates with update utility, which isn’t verified.

• Baselines differ in their methodology for combining planning and return losses and only four seeds are used for the main curves without statistical tests, the main figure gains can’t be attributed to the allocation rule, and small differences can’t be judged without statistics.

Questions for authors

• Beyond the η and κ sweeps, how does SPG respond to continuous or soft-threshold totals?

• Can you add baselines that use the complementary objective with other weighting rules?

• What is the wall-clock overhead of SPG relative to RIS and SPReD and how does it scale with K and batch size ?

2 comments

Matched allocation controls and learning efficiency

Thank you for recognizing the batch-coupled formulation and its proportional-fairness and information-projection interpretations. We provide matched learning controls and seed-level statistics to make the empirical contribution of allocation explicit.

1. Other weighting rules under the same complementary objective.

The Appendix C.2 study contains 40 completed runs: four allocation rules, two layouts, and five training seeds. Every arm uses the same learner, full planning-reference construction, and complementary actor objective. With E={i:Si>0}E=\{i:S_i>0\}, n=∣E∣n=|E|, and C=∑i1{Si≥m}C=\sum_i\mathbf{1}\{S_i\geq m\}, the rules are:

RuleGuidance coefficients
UBC/B on the whole batch
UEC/n on E, zero outside E
FEA development-calibrated constant q0 on E
SPGmin(1,lambda*S_i), with sum_i q_i=C

UB, UE, and SPG share the rule defining C. FE has total q0*n. Each rule is evaluated on its own evolving training batches. The table reports mean ± sample SD over five seeds for normalized joint AUC, with joint=min(random,farthest) computed at each checkpoint before integration.

AllocationAntU joint AUCAntFg joint AUCLayout-equal AUC
SPG0.4286 ± 0.15130.2636 ± 0.08510.3461 ± 0.1006
UE0.3698 ± 0.05580.2424 ± 0.08090.3061 ± 0.0618
FE0.4290 ± 0.12730.1403 ± 0.16790.2846 ± 0.1222
UB0.1048 ± 0.07360.0464 ± 0.10330.0756 ± 0.0729

The AUC windows are 50K-150K for AntU and 50K-400K for AntFg. Layout-equal statistics average the two layouts within each seed. These controls are a separate five-seed study, reported separately from the four-seed main curves.

SPG has the highest layout-averaged mean AUC, improving on UE by 13.1% and FE by 21.6%. Relative to UB, the layout-averaged paired difference is +0.27051, positive in all five seed blocks, with a paired 95% t interval of [0.11252,0.42850]. These comparisons directly demonstrate a learning effect of allocation under a common objective: an additive-versus-complementary loss change cannot explain the differences within this table. The UE/FE comparisons are stated as observed mean improvements, with their seed variation reported explicitly.

2. Why UE is a strong control.

UE retains support-based eligibility, the adaptive total rule, the complete reference, and the complementary objective. Its strong performance is therefore informative: these shared mechanisms already make effective use of planning evidence. SPG adds a support-dependent division of that total and the envelope qi≤min⁡(1,Si/m)q_i\leq\min(1,S_i/m), which prevents weakly supported contexts from receiving a large weight solely because other contexts have strong support.

The random-goal decomposition also shows where the additional benefit appears:

LayoutRandom AUC, SPG / UEInteractions to random 80%, SPG / UEReduction
AntU0.83110 / 0.7968096K / 103K6.8%
AntFg0.84523 / 0.80303155K / 188K17.6%

Random AUC improves in four of five paired seeds in each layout. On AntFg, four seeds reach 80% earlier and one ties. The threshold statistic is the first observed crossing on the 5K evaluation grid. These results support a finite-budget learning-efficiency benefit even when random-goal terminal performance is near saturation.

Fig.5(c) provides a separate task-level allocation test under shared reference construction and a complementary objective:

Fig.5(c), three seedsSPGUniformShuffled
Reported fixed-goal success84.0%56.3%38.0%

Shuffling preserves the coefficient multiset within a batch while changing its assignment. Together, these controls show that the allocation question has measurable consequences for learning.

Finite evidence, continuous totals, and computational cost

3. Why use support counts, and what do the guarantees establish?

Support counts measure how broadly the planning reference is favored over the current policy across sampled candidates. At fixed actions, they are invariant to positive affine transformations of the gate critic. A single candidate with a large estimated value gap does not by itself increase the guidance budget more than a single favorable comparison. The target twin minimum provides conservative evidence, while retaining the complete mixture avoids turning a noisy comparison into selection of one imitation target.

The threshold determines the total, not a binary actor-branch switch: contexts with 0<S_i<m can still receive partial guidance. Theorem 1 establishes qi≤min⁡(1,Si/m)q_i\leq\min(1,S_i/m) and bounded influence of one changed comparison for every support-count vector. These allocation guarantees do not assume that support is a calibrated estimate of downstream update utility. Proposition A.2 separately identifies the sufficient condition for a favorable optimizer-level comparison: support-associated utility must exceed the mismatch and finite-step terms. The matched learning controls in our first response test the allocation's practical effect under that distinction.

4. Continuous totals.

The capped proportional solver and the information-projection characterization allow real-valued totals. We examined a sigmoid centered at m-1/2, normalized to define C_tau=sum_i h_tau(S_i), where h_tau(0)=0 and h_tau(K)=1, with K=16 and m=4. Holding each support vector and the full allocation rule fixed, we recomputed weights on 20 archived UE donor batches:

Total ruleMean C/BMean absolute change per q_i
Hard threshold0.723830
Soft, tau=0.50.723330.00143
Soft, tau=1.00.715450.00838

These are paired static operator measurements. They demonstrate that continuous totals can be integrated without changing eligibility or the proportional solver, with small local coefficient changes on the evaluated states. More generally, the support envelope is retained whenever C_tau<=sum_i min(1,S_i/m); both tested constructions satisfy this condition on the evaluated support domain. Thus the framework accommodates smooth totals, while the hard count has the particularly transparent interpretation that each sufficiently supported context contributes one unit to the total.

5. Training cost and scaling.

We profiled synchronized full learner updates on an NVIDIA A800-SXM4-80GB in FP32, using three real replay batches and three blocks of ten updates. The default matched cost control retains the full K=16 planning reference and both actor-loss terms, fixing q=0.5.

ConfigurationUpdate time, msPeak allocated GPU memory, GiB
Full-reference fixed-q control13.11 ± 0.210.15813
SPG, K=16, B=204817.40 ± 0.130.15886

SPG adds 4.30 ms per update (32.77%) relative to this matched control and approximately 0.74 MiB of peak allocated memory. The allocator itself takes 0.447 ms, about 2.6% of the full SPG update; the separately instrumented evidence call takes 3.122 ms. This directly quantifies the cost of this implementation under a shared reference. These timings are relative to the stated fixed-q control.

KBFull update, ms
4204814.98 ± 0.21
8204816.05 ± 0.30
16204817.40 ± 0.13
32204820.89 ± 0.87
1651215.90 ± 0.48
16102416.59 ± 0.49

The reported SD is across timing blocks. Cached-batch timings exclude environment interaction, replay sampling, batch transfer, and evaluation. The allocation requires candidate evidence evaluation and a sort-based water-filling solve; these measurements characterize its practical scaling over the tested K/B range. At K=4 and eta=0.25, m=1 is a special fully saturated eligible case.

Together with the matched-objective learning controls, these results give a concrete account of what SPG adds: measurable improvements in finite-budget learning, a support-constrained allocation with explicit guarantees, compatibility with continuous totals, and quantified implementation cost. SPG adds no learned allocation network or separate deployment module.

Reviewer_iVMyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Support-Proportional Guidance (SPG) for sparse-reward goal-conditioned reinforcement learning. For each replay context, it counts planning-induced actions whose target-critic values exceed the value of a sampled policy action. The number of contexts passing a support threshold determines the total guidance coefficient. This total is distributed over contexts with positive support through capped proportional allocation. The resulting weights combine return learning and KL guidance with complementary coefficients, while retaining the complete subgoal-conditioned action mixture as the reference. The paper proves a support-dependent weight bound and bounded sensitivity to a single comparison change, and gives a conditional comparison with uniform allocation. Experiments report faster learning on six navigation/manipulation tasks with RIS-derived references, additional gains with graph-derived references, and navigation ablations of guidance strength and allocation.

Critical strengths
  1. Focusing on sparse-reward goal-conditioned reinforcement learning, this paper uses planning references to support policy learning. Since both the planner and the policy evolve throughout training, appropriately controlling planning guidance is a meaningful problem.

  2. Experiments with learned subgoal references from RIS and graph-based planning references from PIG provide preliminary evidence that the method is applicable beyond a single planning source. Moreover, SPG primarily operates during training and introduces no additional SPG inference module at deployment, facilitating integration with existing planning and policy learning systems.

Critical weaknesses
  1. Existing ablations do not adequately distinguish whether the performance gains arise from the complementary loss formulation, differences in sample weights, or dynamic adjustment of the total guidance budget. Compared with the original RIS, SPG changes both how the reinforcement learning and planning guidance losses are combined and how the guidance weights are computed. Uniform and Shuffled controls mainly support the importance of matching weights to samples, but do not independently establish the necessity of an adaptive total guidance budget.

  2. Guidance is based on the number of planning candidate actions that outperform the current policy action. However, whether a higher count translates into greater actual benefit from KL guidance toward the full mixture distribution remains directly unverified.

  3. Some implementation details and comparison conditions are insufficiently specified, limiting reproducibility and the assessment of fairness. For example, the exact weighting formula used in the adapted SPReD is not adequately described. Details about the construction of the Oracle critic, including its data sources and training conditions, are also unclear.

  4. The paper lacks a quantitative analysis of training efficiency and computational overhead. Although the method introduces no new inference module, this does not establish that its training cost is low. The paper also does not report per-update runtime, total training time, or wall-clock time to reach a matched success rate under the same planning-reference conditions.

Questions for authors
  1. Could you provide matched comparisons between the additive and complementary loss formulations to distinguish the benefits of adapting the total guidance budget from those of changing the loss formulation?

  2. Is there independent evidence that higher support predicts greater benefit from replacing the return gradient with the KL gradient toward the full reference distribution? Could this be verified by comparing updates from identical training states and evaluating them using independent rollouts or return estimates?

  3. How is the Oracle critic obtained, and what additional data or privileged information does it use? Do the configuration changes in Figure 4(c) also affect return optimization and value learning?

  4. Relative to baselines using the same planning references, how much additional per-update runtime, total training time, and GPU memory does SPG require? When reaching the same success rate, do the savings in environment interactions translate into reductions in actual training time?

2 comments

Thank you for recognizing the importance of controlling an evolving planning signal and the value of testing two planning sources. We address the attribution question with four allocation rules evaluated under shared planning-reference construction and a complementary actor objective.

Matched allocation controls and the role of support evidence

1. Allocation has a measurable effect when the loss formulation is held fixed.

In the five-seed study of Appendix C.2, all four arms use the same full-mixture reference construction, critic configuration, optimizer, and complementary objective. Let E={i:Si>0}E=\{i:S_i>0\}, n=∣E∣n=|E|, and C=∑i1{Si≥m}C=\sum_i\mathbf{1}\{S_i\geq m\}. The rules are:

RuleGuidance coefficients
UB: full-batch uniformq_i=C/B for every context
FE: fixed eligible rateq_i=q_0 on E; zero outside
UE: eligible uniformq_i=C/n on E; zero outside
SPGq_i=min(1,lambda S_i), with sum_i q_i=C

The table reports normalized joint AUC, computed from the checkpointwise minimum of random- and farthest-goal success. Each layout uses five matched training seeds; values are mean ± sample SD. The windows are 50K-150K for AntU and 50K-400K for AntFg. Layout-equal statistics average the two layouts within each seed. These runs are separate from the four-seed main curves.

AllocationAntU joint AUCAntFg joint AUCLayout-equal AUC
SPG0.4286 ± 0.15130.2636 ± 0.08510.3461 ± 0.1006
UE0.3698 ± 0.05580.2424 ± 0.08090.3061 ± 0.0618
FE0.4290 ± 0.12730.1403 ± 0.16790.2846 ± 0.1222
UB0.1048 ± 0.07360.0464 ± 0.10330.0756 ± 0.0729

SPG has the highest layout-equal mean, improving it by 13.1% over UE and 21.6% over FE. Against UB, the paired layout-equal AUC difference is +0.2705, positive in all five seed blocks, with a 95% paired t interval of [0.1125,0.4285]. These differences are obtained without changing the actor loss formulation, directly supporting a contribution from allocation.

UE is a strong control: it already retains support-based eligibility and the same rule for setting C. Its performance shows the value of these shared mechanisms; SPG further makes each share depend on support strength. For a given support vector, SPG and UE have identical total mass. During separate training runs, each recomputes that rule on its own batches.

FE keeps uniform eligible shares but replaces C/n by a layout-specific constant q_0; its total is q_0*n, not a fixed C. On AntFg, UE and SPG improve mean AUC over FE by 0.1021 and 0.1234, respectively. On AntU, SPG and FE are nearly tied. Thus, the control supports the practical value of coordinating total strength and allocation across the evaluated layouts, without conflating a fixed eligible rate with a fixed-total proportional rule.

The separate fixed-goal experiment in Fig.5(c) reports the following under the common objective:

Fig.5(c), three seedsSPGUniformShuffled
Reported fixed-goal success84.0%56.3%38.0%

Shuffling preserves the coefficient multiset within a batch. This tests whether the assignment of guidance to contexts matters, beyond using unequal coefficients. We keep its three-seed protocol separate from C.2.

2. Support guarantees and update utility.

Theorem 1 holds for every support-count vector: qi≤min⁡(1,Si/m)q_i\leq\min(1,S_i/m), and a single changed comparison has bounded influence on the entire allocation. These properties do not require a support-utility correlation assumption. For example, at m=4, one favorable comparison permits at most 0.25 guidance weight; UE has no corresponding support-dependent cap.

Proposition A.2 addresses the distinct learning question. It compares the actual optimizer step from an identical state and states explicitly when the support-associated gain exceeds residual error and finite-step curvature. Appendix C.5 tests coefficient and optimizer sensitivity with fixed references and actor noise; the task controls above test learning outcomes. These are different from directly calibrating support against an independent rollout-based update utility. The structural guarantees and common-objective learning evidence stand on their own, while the conditional proposition states the additional requirement for a local improvement guarantee.

Training cost and implementation details

3. Measured learner cost and sample efficiency.

We profiled the full learner update on one NVIDIA A800-SXM4-80GB with FP32, K=16, and B=2048. The matched-reference cost control retains the full K-component mixture and both actor loss terms, with q fixed at 0.5. Results are mean ± sample SD across three blocks of ten updates using three archived replay batches.

ConfigurationUpdate timePeak allocated memoryPeak reserved memory
Full-K, fixed-q control13.11 ± 0.21 ms0.15813 GiB0.18555 GiB
SPG17.40 ± 0.13 ms0.15886 GiB0.18555 GiB

SPG adds 4.30 ms per update (32.77%) and approximately 0.74 MiB of peak allocated memory relative to this control. Separate instrumentation measures the support-evidence call at 3.122 ms and the allocator at 0.447 ms. The latter is about 2.6% of a full SPG update. These measurements include learner computation on cached batches; environment stepping, replay sampling, batch transfer, and asynchronous evaluation are outside the timing window. The reference-matched control isolates the cost of adding support evaluation and allocation; the table reports this comparison specifically.

The same five-seed allocation study also shows a practical sample-efficiency gain over the stronger UE control:

Random-goal metricAntU UEAntU SPGAntFg UEAntFg SPG
Normalized AUC0.79680.83110.80300.8452
Interactions to first observed 80% success103K96K188K155K
Success at the training budget99.8%100.0%99.4%99.4%

The mean interaction reduction is 6.8% on AntU and 17.6% on AntFg. On AntFg, SPG reaches the threshold earlier for four paired seeds and ties for one. These are random-goal decompositions of the C.2 study, evaluated every 5K interactions; the joint-AUC results are reported in our preceding response. The cost table quantifies update overhead, and this table quantifies sample efficiency. We report these as per-update cost and interaction efficiency, respectively. SPG's deployment benefit is that it adds no SPG allocation module to action selection.

4. Network roles and reference construction.

In the four-arm implementation, the target twin critic supplies support evidence, the online twin minimum supplies the actor return term, and the target twin minimum supplies TD bootstrap values. The TD next action comes from the online actor. Evidence aggregation is the mean minus one population standard deviation across two heads, which equals their minimum in exact arithmetic. Both online critic heads regress the same TD target. The actor uses the continuous, detached coefficients in B−1∑i[qiα KLi−(1−qi)Qonline,i]B^{-1}\sum_i[q_i\alpha\,\mathrm{KL}_i-(1-q_i)Q_{\mathrm{online},i}].

All K=16 reference components remain in the mixture density; support does not select a single critic-preferred action or truncate the reference. Reference construction, evidence estimation, and return optimization are distinct roles. The four-arm allocation experiment holds these roles fixed, allowing its learning differences to be interpreted without changing the loss or critic architecture.

The submitted Oracle comparison is defined as replacing the support-evaluation critic while retaining the planner, allocation rule, full reference, and online return critic. We use it as a privileged evidence-quality diagnostic; the task-performance claims above concern learned-critic SPG. Likewise, Fig.4(c) should be read as a critic-configuration comparison; the four-arm study is the direct control for allocation attribution. The common-objective task controls and measured implementation cost provide the direct evidence for the contribution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.