Goal-conditioned RL commonly uses hindsight relabeling to associate each state-action pair in a replay buffer with a goal reached later in the same trajectory. By mixing experience collected under different commanded goals, this implicitly samples from a distribution over goals that is rarely made explicit. We formalize it as the posterior-averaged goal occupancy measure and introduce Kerl, an actor-critic method grounded in its kernel mean embedding. With a characteristic kernel, the kernel mean embedding uniquely determines the underlying distribution. In practice, Kerl approximates it in finite dimensions by training the critic to predict the expected random Fourier features of future goals. These fixed features replace the learned goal encoder of contrastive RL, while a squared-error regression loss replaces its contrastive objective, reducing computational cost within the same actor-critic structure. The learned representation enables reconstruction of the goal distribution, revealing which goals are expected to follow a given state-action pair. Our experiments and ablation studies show that Kerl scales effectively with network depth, achieving large gains on the hardest JaxGCRL tasks.
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.Reviewer_5L6xOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
1
Soundness
3
Read review
Paper summary
The authors propose the posterior-averaged goal occupancy measure as a representation learning signal to train a goal-conditioned policy where (state, action) pairs and goals live in the same latent space. Then they approximate the kernel mean embedding of that space with random Fourier features and introduce KERL, an actor-critic method grounded in the resulting feature space that benefits from its favorable representational properties. They show that KERL scales effectively with network depth, achieving large gains on the hardest JaxGCRL tasks.
Critical strengths
the paper addresses a relevant goal space representation learning question under a quite original perspective
the paper presents a clear conceptual contribution by explicitly formalizing the implicit distribution behind HER
the method replaces an unstable Contrastive RL method with a simpler least-squares regression problem with a fixed latent space. It is simple, as it adds minor changes to CRL (a fixed feature map and an MSE loss), and has large gains on tasks where prior work fails
the ablations are informative as well
Critical weaknesses
despite the obvious didactical writing and illustration effort, Sections 2, 3 and 4 are still hard to follow for a non mathematically-oriented reader
the proposed KERL method changes two things about CRL: a fixed goal encoder (which is learnable in CRL) and regression loss (replacing the InfoNCE loss in CRL), so the reason KERL surpasses CRL is unclear and untested.
KERL is only demonstrated with low-dimensional goals, which are all coordinates.
Questions for authors
could you determine which is the most important source of improvement in KERL performance?
can the fixed goal encoder used in KERL be adapted to goals of high dimensions such as visual inputs, which is a setting used in CRL?
if I understood correctly, with the green versus red/white circles in the top-right corners, Fig. 3 provides evidence that the posterior-averaged goal occupancy measure provides a better learning signal than the more standard value function. Is this correct? If so, what makes you believe that the effect observed in Fig. 3 does not depend on the chosen toy environment? Couldn't we get the reverse conclusion with a different environment?
any idea why SRL outperforms KERL in Ant Hardest Maze, but not in the other environments?
more broadly, do you have the beginning of an explanation why KERL outperforms CRL and SRL in practice? On the hardest environments where KERL outperforms CRL and SRL, could you track a representational error that CRL and SRL are making in a greater extent than KERL?
what is the goal space for arm-binpick-hard? It is not stated in the paper.
We thank the reviewer for their positive assessment and careful reading. We are glad they found our perspective original, the formalisation of the distribution behind hindsight relabelling a clear conceptual contribution and our method a simpler alternative to CRL with large gains.
We also thank the reviewer for acknowledging our effort on the writing and illustrations. We agree that the paper takes time to digest and that the theoretical parts can be challenging for some readers from different background. We therefore tried to write each section so that its intuition and results can be followed without reading the proofs, and supported them with figures. In the revision, we added rendered frames of the agents (Sec. 5.5, Fig. 8 and App. E.7 Figs 24, 25 and 26), which show how KERL's behavior changes as depth increases. The general answer at the top of the rebuttal lists all the new content.
We also appreciate the reviewer's questions. The question of which change drives KERL's improvement over CRL, the fixed goal encoder or the regression loss, led to a new ablation that separates the two (Sec. 5.6). We believe it substantially strengthens the paper. The questions on high-dimensional goals, on whether Fig. 3 generalises beyond its toy environment, and on why KERL outperforms CRL and SRL also helped us clarify the scope of our claims. We answer each question below, in the order the reviewer asked them.
Q1. The main source of improvement (Fig 9 in the revised paper, depth 16, full tables in our reply to aVMR)
Critic
Ant Big Maze
Humanoid
Ant Hardest Maze
KERL: fixed RFF + regression
730 ± 18
743 ± 7
607 ± 66
fixed RFF + InfoNCE
494 ± 38
95 ± 19
329 ± 42
frozen random network + regression
55 ± 35
20 ± 1
3 ± 3
CRL
424
522
392
Neither change alone explains the gain; the combination does.
On the humanoid mazes the gap is larger: CRL with KERL's fixed features scores 64 and 17, against 569 and 368 for KERL (reply to hztU, Table H1 in the rebuttal).
Depth is the second ingredient, and it must go into the critic: 15 → 811 on Humanoid by deepening only the critic (Fig. 6; reply to aVMR).
The implicit dense reward does not explain it alone: SAC on that reward is 0 at depths 8 and 16 (Fig 10 in the paper, Table H2 in the rebuttal).
Q2. High-dimensional goals such as images. All our goals are coordinates:
Early intuitive response: new Prop. 3.1 does not depend on the goal dimension, which justify the use of random Fourier features for control (but this remain only a theorical result)
We fully agree with the reviewer that evaluating Kerl on visual-based settings would further validate its strength, we leave this for futur work and add it in the limitation section.
Q3. Does Fig. 3 show that the posterior-averaged occupancy is a better signal than the value function? No. Fig. 3 is an illustration, not evidence that the posterior-averaged occupancy is a universally better learning signal than the standard Q-function. Its purpose is to show how the two differ, why the occupancy can be used directly for policy evaluation, and why it can be more informative than the Q-function in some situations. The reviewer is right that a different environment might reverse the conclusion, and the revision now explicitly limits the claim to the example ("In this example, the Q-function seems a poor ..."). Establishing theoretically when each provides the better learning signal is an important but challenging open question.
Q4. Why is SRL ahead on Ant Hardest Maze? SRL and KERL reward different things. SRL learns to minimise the first hitting time, the time needed to first reach the goal. As the SRL authors note (p. 6), this pushes the agent to reach the goal as fast as possible with no incentive to slow down, which favours bang-bang control. On a stable body such as the ant, this is not harmful and can even help: the ant runs at full speed, overshoots, comes back and stays, so it arrives early and still accumulates a high time at goal. This is where SRL is ahead (Ant Hardest Maze, [SRL 760 vs. KERL 665]).
On an unstable body such as the humanoid, the same greediness is harmful. The agent lunges at full speed and falls on arrival (SRL, p. 7), which leaves little time at goal. The SRL authors introduce a dwell-goal formulation for this reason, and it only partly fixes the problem.
KERL instead targets the posterior-average occupancy of the goal in the future: its objective scores how much time the agent will spend at the goal, not only whether it gets there. Staying stable at the goal is therefore part of the objective itself. This matches the results: KERL is close to SRL on the ant mazes, and far ahead on the humanoid and manipulation tasks, where not overshooting matters.
Q5. Why does KERL outperform CRL and SRL? Representational errors?
Empirically, the ablations in Q1 show where KERL's gains come from.
CRL. We believe KERL's main advantage over CRL is the fixed goal representation combined with a regression loss. CRL must learn the state–action encoder and the goal encoder jointly, which makes learning harder and less stable, and the contrastive loss adds a further difficulty, as described by Wang & Liu (2021).
SRL. As discussed in Q4, we believe the limitation lies mainly in its objective, which targets reaching the goal rather than staying at it.
Q6. Goal space of Arm Binpick Hard. The 3-D position of the cube. We will state it in the paper.
Wang, Kevin, et al. "1000 layer networks for self-supervised rl: Scaling depth can enable new goal-reaching capabilities." Advances in Neural Information Processing Systems 38 (2026): 157643-157670.
Wang, F., & Liu, H. (2021, June). Understanding the behaviour of contrastive loss. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 2495-2504). IEEE.
I thank the authors for the insightful rebuttal and the clear answers to my questions. I will wait for the discussion with the other reviewers and AC before eventually upgrading my score. In between, could the authors highlight in the revised version the changes they have made wrt the original version (e.g. all changes in blue)? That would help reviewing them without re-reading the whole paper again...
Acknowledgement and PDF-diff now in supplementary material
We are very grateful for the quick reply and that the reviewer is satisfied with our rebuttal and our answers.
We thank the reviewer for suggesting to add a PDF-diff, we agree that it is very relevant and helpful. We have added it as supplementary material. We kept it separate from the main PDF because the diff might make the paper harder to read.
The paper names the distribution that hindsight relabelling samples future goals from, the posterior-averaged goal occupancy ρπ(⋅∣s,a), and builds KERL around its kernel mean embedding. The critic regresses onto fixed multiscale random Fourier features (RFF) of relabelled future goals, so that μω(s,a)≈E[z(G+)∣s,a]. The actor scores actions by the inner product or the squared distance between μω(s,a) and z(g). Relative to CRL, this removes the learned goal encoder and replaces InfoNCE with squared-error regression. On JaxGCRL, KERL improves with depth, mainly through the critic, and makes large gains on the humanoid mazes. Alg. 2 reconstructs ρπ from μω for visualisation.
Critical strengths
A simplification of CRL turns the humanoid big and hardest mazes from failures into partial successes. On humanoid-big-maze, scaled CRL and SRL reach time-at-goal 59 and 45; KERL reaches 400. On humanoid-hardest-maze, both baselines reach 2; KERL reaches 340, at equal wall-clock (Fig. 2). KERL removes components rather than adding them. A gain of this size from a simplification deserves an explanation in its own right, whatever it turns out to be (see weakness 1).
The paper states precisely what the critic estimates, and what it does not. Prop. 4.1 identifies the inner-product score as a posterior mixture of Q-functions for the reward rg(s)=(1−γ)k(φ(s),g), and Eq. 16 adds the self-similarity term of the MMD score. App. B.3 shows that the critic is generally not the Q-function of the state-wise average policy, which the paper presents as correcting Lemma 4.1 of Eysenbach et al. (2022). Because the features are fixed and known, μω(s,a) estimates an explicit moment vector that can be checked. Much of the analysis below is possible only because of this precision.
Depth pays off in KERL's critic rather than its actor. With a 4-layer actor, a 64-layer critic takes humanoid from 15 to 811 and ant-big-maze from 248 to 702. With a 4-layer critic, scaling the actor gives at most 19 and 342 (Fig. 6). On Ant Big Maze this is the opposite of what Wang et al. (2026, Sec. 4.4) report for CRL, where scaling the actor matters more. It is useful evidence about where capacity matters in this family of methods.
Critical weaknesses
The small numerical checks I claim to have performed below are idealised toy computations. If useful, happy to send source cross across upon request, please contact the area chair to coordinate.
I would raise my score to weak accept if the revision (i) restates the theoretical claims as asymptotic and the reconstructions as illustrations unless they are validated, (ii) states the implicit dense reward and which reward each baseline used, and (iii) adds the bandwidth ablation and the CRL with fixed RFF goal features control, at least on the humanoid mazes. These are fixable problems I am confident the authors can address, but on its current state the paper should not be accepted.
Weakness 1. The gains that KERL exhibits cannot be unambigously attributed to the embedding. By Prop. 4.1, KERL maximises the value of a dense Euclidean kernel reward, while the methods it is compared against in Fig. 2 receive no such shaping.
This needs new experiments and a change of framing. Please:
compare against a TD method trained on KERL's own reward rg with the same architecture;
ablate the bandwidth set;
separate the changes relative to CRL;
report per-goal results.
The conclusion's attribution of the gains should also be reworded.
rg(s)=(1−γ)kMS(φ(s),g) averages Gaussians with bandwidths up to σ=100, on mazes 24–49 units across. It is therefore nonzero everywhere and decreases with Euclidean distance to the goal, ignoring walls. The default MMD actor maximises 2K minus a self-similarity term (Eq. 16), so it carries the same dense term. This sits uneasily with the introduction's framing of GCRL as moving "beyond hand-designed rewards", and Fig. 2 compares against SAC+HER, which is trained on a sparse success indicator. If the shaping is intended, that is a legitimate design choice, but then the informative baseline is a method that receives the same shaping.
The paper's own Table 3 suggests that dense rewards matter on exactly these tasks. In the public JaxGCRL code, the humanoid mazes return a dense environment reward: negative distance to the goal, plus a healthy bonus, minus a control cost. The ant mazes default to the sparse success reward, and SAC without HER trains on the environment reward. If Table 3's SAC numbers were produced this way, a dense-reward baseline already beats scaled CRL and SRL on humanoid-U-maze (∼250 vs. 159 and ∼220) and humanoid-big-maze (∼175 vs. 59 and ∼50), while scoring about 0 on the sparse-reward ant mazes. KERL still clearly beats it (516 and 400), so shaping is not the whole story, but it is plausibly part of it. Wang et al. describe their setting as sparse-reward, so please state which reward each baseline used (Q2).
The CRL comparison changes four things at once. Relative to Wang et al.'s public CRL implementation, KERL changes:
the goal encoder, from learned ψ to fixed RFF z;
the critic loss, from InfoNCE to regression;
the actor score, from −∣ϕ−ψ∣2 to −∣μω−z∣22;
the entropy temperature, from learned with target −0.5∣A∣ to fixed α=0.5.
The paper names the first two. The sequence CRL -> CRL with fixed RFF goal features (still InfoNCE) -> KERL, holding the actor score and α fixed, would show which change matters.
The conclusion attributes the gains using the wrong comparison. "Extensive ablations attribute these gains primarily to kernel mean embedding network depth" (Sec. 6) rests on depth ablations within KERL. Sec. 5.2 itself notes that "depth alone does not account for the gap" to CRL.
No experiment varies the bandwidths. Sec. 5.5 states that the bandwidth "is not a free parameter" and fixes eight scales. Dropping the coarse scales (σ≥10), or using a single scale, would test directly how much of the long-range signal is Euclidean shaping. Running the same ablation on the CRL-with-fixed-RFF variant would also show whether the coarse scales are what gives KERL long-range signal at all: KERL's actor scores the smoothed density linearly, whereas the regularised InfoNCE score in Wang et al.'s code approximates a log density ratio, which keeps its gradient where little mass lies near g.
Distance-based rewards predict failures on specific goals behind walls. Rewards that decrease with Euclidean distance have optima that stall behind walls close to the goal (e.g. Trott et al., 2019). In a planar gridworld abstraction of the hardest-maze layout, the optimal policy for rg, starting from the maze start, never reaches two to four (depending on the assumed speed) of the goals in the pocket at humanoid-layout positions (10,8), (12,8), (14,8) and (14,10), i.e. layout cells (5,4), (6,4), (7,4) and (7,5). It stalls on the other side of a wall, 3–5 units from the goal. This concerns the reward, not KERL's posterior-averaged actor; per-goal success rates on the hardest mazes would show whether KERL inherits the failure.
Weakness 2. The theory does not explain why KERL works with 64--1024 features. The cited bounds are vacuous at that size, and control needs a different, dimension-free guarantee from the one reconstruction needs.
This is a writing change. Please:
reword the sentences quoted below;
scope "m=64 is enough for the tasks considered" (Sec. 3.2) to the three tasks on which m was ablated (arm-push-hard, humanoid, ant-big-maze; not the humanoid mazes);
consider replacing Lemma 3.1, as the justification for small m, with a decision-level argument such as the one below.
The first issue is that Lemma 3.1 is vacuous at the m used. With the paper's bandwidths and maze diameters, the constant is h≈261 (ant-hardest, d=2) to 313 (humanoid-hardest, d=3). The 95% bound at m=1024 is then 8.2–9.9, on an error that is at most 2 for sine–cosine features; a bound of 0.1 would need about 7–10×106 features. "These guarantees justify using the finite-dimensional RFF features" (Sec. 3.3) and "concentration bounds quantify approximation error" (Sec. 6) are only proven to hold asymptotically in practice. The lemma's assumption ∫∥ω∥2dΛ<∞ also excludes the Laplace kernel of the ablations, whose spectral density is Cauchy (App. C.1, Table 1), yet "Gaussian, Laplace and Cauchy kernels perform equally well" (Sec. 5.5).
Second, the proof in App. B.4 gives MMDk(ρ^hϕ,ρπ)≤4εm+2(k(0)−k(h)), where εm is the uniform kernel error. That is Op(m−1/4), not "the same convergence rate as K^" (Sec. 4.2). It also concerns the exact RFF embedding μ^π, not the learned μω, and an exact QP solution rather than a finite run of Alg. 2.
Nevertheless, for control, finite features only need to preserve the ranking of actions, and the error in that ranking does not grow with the goal dimension or the size of the domain. Fix two actions whose future-goal distributions are P and Q, and a goal g. Let D=MMDk(P,Q), and let Δ be the exact gap in the score. For cosine-bias features drawn independently of P, Q and g, the RFF estimate of the gap is unbiased, with variance at most 2m3D2 for the inner-product score and m24D2 for the MMD score. By Chebyshev,
P[misrank]≤2mΔ23D2(inner product),P[misrank]≤mΔMMD224D2(MMD).
Neither bound depends on the goal dimension, the size of the domain or any resolution. Whether m=64 suffices becomes a question about the relative gap Δ/D of the decisions the actor faces, which can be measured. The bounds ignore the critic's regression error and the dependence of a trained policy on the features drawn.
Finally, the distribution is not retained at m=64 either. On the hardest-maze layout, a distribution lying entirely inside walls and one lying entirely in free space have identical 64-dimensional feature means (see weakness 3). So "below this threshold, the representation cannot retain the distribution" (Sec. 5.5) does not explain the collapse at m∈8,16: at m=64 the representation does not retain it, and learning works.
Weakness 3. The reconstructions are presented as revealing the distribution the critic has learned. But finite features do not determine that distribution, and the critic's error is never measured.
Please either tone down "fully interpretable" (Sec. 6), "reveals the distribution estimated by the mean embedding" (Sec. 4.2) and "accurate" (Fig. 7), or validate them:
against ground truth where ρπ is known: the App. D tabular MDP and the Fig. 3 point environment;
by reporting the critic-error lower bound below for the state–action pairs plotted.
The first important thing to highlight is that the identifiability of the reconstructed distribution fails at any finite m. To see this, note that the map P↦EP[z] is linear into Rm, so for any m+2 goals the vectors (z(gi),1) are linearly dependent. Splitting a dependence into its positive and negative parts gives two distributions with disjoint supports and identical feature means. App. A's "because characteristic kernel mean embeddings uniquely identify distributions, the represented occupancy distribution can be recovered" therefore holds for the ideal embedding, not for the learned one. On the hardest-maze layout, with distributions uniform on 0.5-unit cells, two distributions with identical 1024-feature means put 73% and 24% of their mass inside walls; with 64 features, 100% and 0%.
Alg. 2's output is therefore one of many consistent distributions, selected by the ridge warm start, the simplex constraint and the stopping time. The difference between the 4- and 64-layer reconstructions in Fig. 7 is not shown to reflect the critics rather than this selection, and μω is never compared with the embedding it targets.
However, the critic's error can be bounded below without ground truth. For every c∈Rm with ∥c∥≤1 and every distribution ν on the goal space, ∥μω(s,a)−Eνz∥≥c⊤μω(s,a)−supxc⊤z(x); maximising over c gives a lower bound with no discretisation of the distribution. In a toy check, it recovered 0.047 of an error of norm 0.063 (10% of ∥μ∥, in a random direction). It cannot detect an error that lands on another valid embedding.
In that vein, region masses can be bounded in the same way, but usefully only for very accurate critics. If e≥∥μω−Eρz∥, then for any region W and every c, ρ(W)≤c⊤μω+e∥c∥+supx(1W(x)−c⊤z(x)), and symmetrically from below. In the toy, for a clean corridor occupancy they certify at most 8.8% wall mass from an exact 1024-feature embedding, but only 22% and 31% at critic errors of 1% and 2% of ∥μ∥. For an occupancy with 15% of its mass on a shortcut through a wall, they certify only 5%: mass at a wall boundary can be moved across it at almost no cost in feature space.
Finally, it is worth remarking that in the humanoid reconstructions, mass over a wall is not by itself an error, but the settings needed to interpret the figures are missing. Humanoid goals are 3D torso positions, and the walls in the humanoid mazes are waist-high: they reach a height of 1.0, while an episode terminates if the torso leaves the 1.0–2.0 band. To my knowledge, humanoids can in principle cross walls, and Wang et al. (2026, Sec. 4.3) report a deep CRL agent vaulting a wall in Humanoid U-Maze. Wall mass is thus unambiguous evidence of error only for the ant mazes. Please state whether your humanoid policies ever cross walls and how the height coordinate enters Alg. 2, and report Alg. 2's domain, number of cells, iteration count and step size.
Weakness 4. Nothing guarantees that the actor update improves the policy. In an idealised version, the posterior-averaged criterion has a strongly suboptimal fixed point.
I don't expect the authors to resolve this. But it should be stated as a limitation, and the paper should not present the posterior-averaged occupancy as the right criterion for comparing actions without qualification. App. B.3 shows that K is generally not the Q-function of the state-wise average policy, which removes the usual improvement argument, and no other is given.
A minimal counterexample: Take deterministic dynamics in which the target can be reached in 4 steps through a hub shared with 44 other commanded goals, or along an exclusive chain of T1 steps. Let the critic be the exact posterior-averaged occupancy with a delta kernel, and the actor ε-greedy. With γ=0.99 and a uniform initial policy, the iteration sends the target along the exclusive chain unless T1≥383 (inner product) or T1≥429 (MMD). Under the MMD actor, the policy that uses the exclusive chain remains a fixed point for every T1 tested, up to 104, whereas value iteration on the success reward picks the hub for every T1>4. The mechanism: the hub's posterior-averaged future is dominated by the other goals, and MMD's self-similarity term, which "favors dispersed actions" (Sec. 4.1), then prefers the long chain.
This is the criterion Fig. 3 promotes. Sec. 2 says that "what really distinguishes the two actions is not how well each reaches g∗, but which goals each leads to in the first place". The example shows that this can be the wrong thing to optimise. A maze with a shared bottleneck would test whether it matters in practice.
Weakness 5. The comparative claims are not scoped to the regime tested, and the limitations paragraph omits the points above.
This is a writing change, plus one targeted experiment if possible.
In the regime tested, dynamics are deterministic, and training is online and data-rich. In the mazes, commanded goals are drawn from finite sets of cell centres (6, 25 and 45 in the U-, Big- and Hardest-maze layouts), and the evaluation goal set is not stated (Q3).
Notably, stitching is not tested. KERL has no Bellman backup, so whether it composes behaviour across trajectories is an empirical question. Wang et al. (2026, Sec. 4.5) report partial stitching for CRL with depth. Training only on nearby goals and evaluating on far ones, as they do, would be a cheap test.
On wording, "Matches or improves the state of the art" (Secs. 1 and 5.1) sits awkwardly with Fig. 2 and Table 3, where SRL has higher reported means on all four ant mazes (e.g. ∼760 vs. 665 on ant-hardest-maze, ∼800 vs. 692 on ant-u5-maze). The "7 of 10 tasks" in Sec. 6 is the accurate summary.
On limitations, the paragraph in Sec. 6 should state the implicit dense reward; that the concentration bounds are asymptotic only; that the reconstructions are not identified at finite m and the critic's error is unmeasured; that no improvement guarantee is given; and the regime above.
Some other minor remarks:
Eq. 13 prints the inner-product score as ⟨μω(s,a),z(g)⟩2, which is inconsistent with Eq. 9 and Prop. 4.1. Which form was used in the ablation?
Lemma 3.1's statement should name the feature map its proof covers (sine--cosine, Eq. 6), as the text after it does.
"170x" means 340 vs. 2. Prefer absolute values.
For humanoid-hardest-maze, please give the baselines' depth and environment steps in Figs. 2 and 4. At equal wall-clock, KERL ran 800M environment steps and CRL 500M (Sec. 5.2).
Questions for authors
Which goals does the actor train on: relabelled future states (as in Wang et al.'s public code), commanded goals, or both? Alg. 1 (lines 10–11) and Eq. 12 leave this open. It determines which part of the kernel the actor's signal comes from (W1, W2).
Which reward did each baseline in Fig. 2 and Table 3 use, in particular SAC on the humanoid mazes (W1)?
Was evaluation on the training goal set or on the *_eval subsets of the public code? How does KERL do on goals sampled off the lattice of cell centres?
Response to Reviewer hztU (1/6): overview and the three conditions
Part 1, overview
We are deeply thankful to the reviewer for an exceptionally constructive review, which we believe helped us substantially strengthen our work. We apologize for posting our response close to the end of the discussion period: addressing all questions carefully led to a lot of new content. We will acknowledge the reviewer's contribution in the final manuscript, and we thank them again for their insightful review and suggestions. Concretely:
Theory. We introduce a new Prop 3.1 inspired by the reviewer's decision-level argument, sharpening it: a one-sided (Cantelli instead of Chebyshev) inequality gives a tighter, dimension-free bound on the probability of misranking two actions. We moved Lemma 3.1 to the Appendix to make space for it (now Lemma B.1). Prop 3.1 uses the sine-cosine features and is proved App B.1, Prop B.1 is the cosine-bias counterpart, proved App B.2. See part 4/6 below for more details.
Behaviors. The reviewer's question on whether our humanoid agents cross walls prompted us to analyze how the behavior of the agents changes with depth. This is now Sec. 5.5 (Fig 8) and App. E.7 (Figs 24, 25 and 26) on various tasks.
Ablations. The requested ablations are in Sec. 5.6 (the whole page 9 is new): CRL's InfoNCE loss with fixed RFF goal features, the bandwidth ablation, and SAC / SAC+HER trained on KERL's own reward (Fig 9, 10, 27 and 28). The humanoid-maze results of CRL with fixed RFF goal features are in App.E.8, Fig 31.
The three conditions the reviewer asked for to increase their score.
Condition
Where it is addressed
(i) theory stated as asymptotic, reconstructions as illustrations
Prop. 3.1 replaces Lemma 3.1 (part 4); reconstructions reframed in Sec. 4.2 (part 5). The word "asymptotic" used throughout the paper.
(ii) implicit dense reward, and the reward of each baseline
Prop. 4.1, Sec. 5.6 (Fig 10), Table 2; SAC and SAC+HER on rg and on the sparse reward (part 2)
(iii) bandwidth ablation + CRL with fixed RFF, on the humanoid mazes
We answer the weaknesses in order: W1 in parts 2–4, W2 in part 4, W3 in part 5, W4, W5 and the minor points in part 6. Q1 and Q2 are answered in part 2, Q3 in part 6.
Response to Reviewer hztU (2/6): W1, the dense reward and the changes relative to CRL
Part 2: W1, the dense reward and the changes relative to CRL
The reviewer is right that, by Prop. 4.1, KERL implicitly values the dense reward rg=(1−γ)k(φ(s′),g); the revision states it (Prop. 4.1, Sec. 5.6, limitations). We address the four requested items in turn: a TD method on rg (H1, below), the changes relative to CRL (below), the bandwidth ablation (part 3) and per-goal results (part 4).
H1. A TD method on KERL's own reward (Humanoid U-Maze, 350M steps, KERL's architecture). "Sparse" is the relabelled success indicator. Fig 10 in the revised paper.
Method, reward
Depth 4
Depth 8
Depth 16
Depth 32
KERL
15 ± 3
329 ± 82
476 ± 62
569 ± 68
SAC, rg (Fig. 10)
61 ± 75
0 ± 0
0 ± 0
0 ± 0
SAC+HER, sparse
0 ± 0
0 ± 0
0 ± 0
0 ± 0
SAC+HER, rg
0 ± 0
0 ± 0
0 ± 0
0 ± 0
Shaping helps a shallow SAC learner on Humanoid U-Maze, and so does rg: SAC reaches 61 at depth 4.
No TD variant scores above 0 beyond depth 4, while KERL climbs to 569.
This matches Wang et al. (2026) (their Fig. 12 and App. A.2, p. 15) for SAC with the dense environment reward (negative distance to the goal, plus a healthy bonus, minus a control cost) and SAC+HER with the sparse reward.
This suggests that the dense signal in KERL's formulation therefore does not explain KERL's gains alone.
Q2. Which reward did each baseline use?
KERL, CRL and SRL use relabelled goals; the reward each one implicitly optimises is now given in Table 2, together with SAC+HER and SAC reward.
SAC+HER in Fig. 2 and Table 4 uses the sparse success indicator.
SAC without HER in Table 4 is copied from Wang et al.(2025). As the reviewer inferred from the public code's defaults, it is trained on the dense environment reward.
The four changes relative to CRL.
Goal encoder and 2. critic loss. The requested sequence CRL → CRL with fixed RFF (InfoNCE) → KERL, with the actor score is in Fig. 9 (depth 16). CRL with fixed RFF reaches 494 ± 38, 95 ± 19 and 329 ± 42 on Ant Big Maze, Humanoid and Ant Hardest Maze, against 730 ± 18, 743 ± 7 and 607 ± 66 for KERL. The humanoid mazes are in part 3.
Actor score. The squared distance in the MMD actor objective is now stated (Sec. 5.4).
Entropy temperature. We apologise for the misleading wording. KERL and CRL tune α identically in our code: α is learned with target entropy −0.5∣A∣, as in the official CRL implementation (train.py, line 560). The initial submission referred to entropy_param, which is fixed at 0.5 in the public CRL code (train.py, line 92). We corrected the manuscript (Table 3).
Conclusion attribution. We reformulate the conclusion to include the contribution of the important components.
Q1. Which goals does the actor train on? The relabelled future goals as in the public CRL and SRL codes. This is now stated explicitly in Sec. 3.4.
Response to Reviewer hztU (3/6): W1, bandwidth ablation and CRL with fixed RFF
Part 3: W1, bandwidth ablation and CRL with fixed RFF
H2. CRL + fixed RFF on the humanoid mazes, with every bandwidth set (depth 32, same step budget as KERL: 350M on U-Maze, 400M on Big Maze). (The learning curves of the first two rows are in Fig 31)
Bandwidths
U-Maze: CRL + fixed RFF
U-Maze: KERL
Big Maze: CRL + fixed RFF
Big Maze: KERL
8 scales, 0.03–100 (default)
64 ± 36
569 ± 68
17 ± 7
368 ± 19
5 fine scales, 0.03–3
97 ± 136
611 ± 97
15 ± 5
341 ± 51
single σ = 0.03
0 ± 0
105 ± 20
0 ± 0
12 ± 10
single σ = 0.1
41 ± 33
496 ± 82
6 ± 3
98 ± 58
single σ = 0.3
137 ± 85
352 ± 273
8 ± 1
271 ± 89
single σ = 1
17 ± 2
146 ± 130
4 ± 2
20 ± 4
single σ = 3
0 ± 0
32 ± 6
0 ± 0
6 ± 1
Fixing CRL's goal features to RFF does not help on these tasks. With the same RFF, InfoNCE fails on both humanoid mazes for every bandwidth set: 64 and 17, against 569 and 368 for KERL( when using the default 8 multiscale kernel). The learning curves (App. E.8) show the gap throughout training: KERL improves steadily, while CRL + fixed RFF stays below about 100 on U-Maze.
The coarse scales do not drive KERL on the humanoid mazes. Removing σ≥10, leaves KERL unchanged within std (569 → 611, 368 → 341). Under InfoNCE, both variants fail with or without the coarse scales.
Multi-scale beats per-task tuning. No single bandwidth matches KERL's default.
H3. Bandwidths on the other three tasks (depth 16, Fig 10 left, Table 5, Fig.27, Fig.28)
Bandwidths
Ant Big Maze
Humanoid
Ant Hardest Maze
KERL, 8 scales (default)
731 ± 17
750 ± 7
611 ± 61
KERL, 5 fine scales
728 ± 35
743 ± 46
476 ± 68
KERL, best single σ
658 ± 53 (σ = 1)
713 ± 62 (σ = 0.1)
517 ± 35 (σ = 1)
InfoNCE, 8 scales (Fig. 9)
494 ± 38
95 ± 19
329 ± 42
InfoNCE, 5 fine scales †
417 ± 40
199 ± 34
251 ± 13
For KERL, the coarse scales matter only on Ant Hardest Maze.
Replacing regression by InfoNCE degrades performance with either bandwidth set.
This work introduces KERL, a goal-conditioned RL method that learns which goals are reachable from a given state-action pair. It formalizes hindsight-based implicit target as a posterior-averaged goal occupancy distribution. This distribution is then represented by kernel mean embeddings using fixed random Fourier features. The critic learns expected future goal features using squared-error regression. The actor learns to select actions by matching the predictive embedding to the test. Empirical results demonstrate strong performance on challenging manipulation and navigation tasks.
Critical strengths
The problem is clearly and well defined. The paper explains and establishes the connection between hindsight goal relabeling, successor and occupancy-based representations, and kernel mean embeddings.
The empirical results provided are sound and convincing (especially on humanoid)
Critical weaknesses
The paper does not independently study the contribution of kernel occupancy modeling from the simpler optimization objective and fixed target representation.
Questions for authors
Can the authors provide studies on the contributions of fixed random Fourier goal features, the squared regression objective, and the kernel-based actor objective evaluated independently?
We thank the reviewer for their positive assessment. We are glad they found the problem clearly defined, the paper easy to follow, and the empirical results sound and convincing. During the rebuttal, we made many new experiments and figures as listed in the general answer, and we will be very grateful if the reviewer has time to take a look.
We are grateful for the question on the contribution of each component of KERL. The study it prompted identifies which components drive the improvement. It is now in the revised paper (Sec. 5.6), and we believe it strengthens the work. Below, we evaluate each component independently :
the fixed random Fourier goal features and the squared regression objective (A1);
the kernel-based actor objective (A2);
the kernel occupancy modelling versus the dense reward it implies (A3).
A1. Fixed RFF targets vs. regression loss. We change one component at a time (Fig. 9; depth 16, MMD actor objective fixed; time at goal, mean ± std, and Fig. 31 as a complement).
Critic
Ant Big Maze
Humanoid
Ant Hardest Maze
KERL: fixed RFF + regression
730 ± 18
743 ± 7
607 ± 66
fixed RFF + InfoNCE
494 ± 38
95 ± 19
329 ± 42
frozen random network + regression
55 ± 35
20 ± 1
3 ± 3
Neither component suffices on its own:
Keeping the fixed RFF targets but training with CRL's InfoNCE loss lowers performance on all three tasks, most sharply on Humanoid.
Keeping the regression loss but regressing onto a frozen random network, a fixed target that is not an approximation of the feature map of a characteristic kernel, fails on all tasks.
This suggests that the gains come from combining fixed kernel features with regression.
A2. Actor objective (Fig. 11 and Fig. 29, last column; depth 64).
Actor score
Ant Big Maze
Humanoid
Arm Push Hard
MMD −∥μω−z(g)∥2 (default)
780 ± 13
799 ± 54
563 ± 148
inner product ⟨μω,z(g)⟩
612 ± 34
735 ± 13
659 ± 54
Neither kernel score dominates on every task. We use MMD by default.
A3. Kernel occupancy modelling vs. the kernel reward it implies. By Prop. 4.1, KERL implicitly values the dense reward rg=(1−γ)k(φ(s′),g). To test whether this reward alone explains KERL's results, we train SAC and SAC+HER directly on rg, with KERL's architecture, on Humanoid U-Maze (Fig. 10, right):
Depth
4
8
16
32
KERL
15 ± 3
329 ± 82
476 ± 62
569 ± 68
SAC on rg
61 ± 75
0 ± 0
0 ± 0
0 ± 0
SAC+HER on rg
0 ± 0
0 ± 0
0 ± 0
0 ± 0
With the dense reward rg:
SAC reaches a non-trivial score at depth 4 but collapses with deeper networks;
SAC+HER fails at every depth;
KERL improves steadily with depth.
These results are consistent with the findings of Wang et al. (2026) on Humanoid U-Maze on SAC with dense reward.
In conclusion: using rg, SAC and SAC+HER do not reproduce KERL's scaling. This suggests that gains come from modelling the kernel mean embedding of the posterior-averaged occupancy, not from the implicit reward alone.
We hope these experiments answer the reviewer's question. If any point remains unclear, or if there is an additional analysis the reviewer would find useful to increase their score, we would be glad to discuss it.
We thank the reviewers for their careful and constructive reviews. We apologize for posting our response close to the end of the discussion period: addressing all questions carefully led to substantial new content (in particular, the whole page 9 is new).
The reviews converged on one question: what in KERL produces the gains? We answer it with controlled experiments. Answering it also led to a new finding: as depth increases, KERL agents develop qualitatively new behaviours, now a dedicated section of the paper (Sec. 5.5).
1) Emergent behaviours with depth (new Sec. 5.5, Fig. 8, App. E.7, Fig.24, Fig.25, Fig.26)
This analysis began with hztU's question of whether our humanoids cross walls. Rendering the agents at increasing depth shows that depth changes how they solve the task, not only how often:
Humanoid U-Maze. The depth-32 and depth-64 agents climb onto the wall instead of walking around it (Fig.8).
Arm Binpick Hard. The depth-64 agent solves the same task in four different ways: carry, fling or bat the cube over both walls, or knock it onto the floor and flick it in (Fig. 8 and App. E.7).
Appendix E.7 (Figs. 24, 25 and 26) shows the same progression across depths on Humanoid, Humanoid Big Maze, Arm Push Hard and Arm Binpick Hard.
2) What produces the gains (Tables A1, H1-H5 are given in the rebuttal response of reviewer aVMR and of reviewer htzU, respectively.)
Question
Experiment
Answer
Where
Fixed features, or regression?
One change at a time, depth 16, 3 tasks
Neither alone; the combination
Fig. 9; Tables A1, H3 in the rebuttal
Is CRL + fixed RFF enough on the humanoid mazes?
Full bandwidth sweep, depth 32
No: 64 ± 36 vs. 569 ± 68 (U-Maze), 17 ± 7 vs. 368 ± 19 (Big Maze)
App. E.8, Fig 31; Table H2 in the rebuttal
Is the implicit dense reward enough?
SAC and SAC+HER on rg and on the sparse reward
No: on Humanoid U-Maze every SAC variant is ≤ 145 and scores 0 at every depth above 4; KERL reaches 569
Fig 10 right; Table H1 in the rebuttal
Does Euclidean shaping from the coarse scales matter?
Drop σ≥10, on 5 tasks
Only on Ant Hardest Maze
Fig 10 left, Fig 28; Tables H2, H3 in the rebuttal
Do goals behind walls fail?
Per-goal success over 45 goals
At depth 64, every goal is reached in ≥ 87% of episodes
Fig 19; Table H4 in the rebuttal
Does KERL stitch?
Ant U-Maze protocol of Wang et al. (2026)
Yes; Better with deeper networks
Fig 20, Table H5 in the rebuttal
3) Theory and presentation (in the revised PDF)
Misranking bound. A new Prop 3.1 was added to Section 3 (to make space, Lemma 3.1 moved to the Appendix, Lemma B.1): dimension-free bound on the probability of misranking two actions (proofs are in App. B.1, and a Prop B.1 is in App. B.2) thanks to reviewer hztU's suggestion.
Implicit rewards of KERL, CRL, SRL and SAC+HER are stated (Table 2).
Limitations now name directions for future work:
quantifying how much the implicit kernel reward contributes;
extending the guarantees to the critic's error and improving the remaining bounds
policy-improvement guarantees for the posterior-averaged actor update;
We hope that the reviewers will have time to look at the main new additions: the new Sec. 5.5 and App. E.7 on emergent behaviours, the new ablations in Sec. 5.6, App.E.4, E.5 and E.8, and the improved theoretical result in Prop. 3.1 and Prop. B.1.
We have added a PDF diff to the supplementary material to help reviewers see what changed. We kept it separate from the main PDF because the diff highlights the many new figures, which might make the paper harder to read. We apologize for not including it in yesterday’s response.
From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.
Response to Reviewer 5L6x (part 1/2)
Part 1
We thank the reviewer for their positive assessment and careful reading. We are glad they found our perspective original, the formalisation of the distribution behind hindsight relabelling a clear conceptual contribution and our method a simpler alternative to CRL with large gains.
We also thank the reviewer for acknowledging our effort on the writing and illustrations. We agree that the paper takes time to digest and that the theoretical parts can be challenging for some readers from different background. We therefore tried to write each section so that its intuition and results can be followed without reading the proofs, and supported them with figures. In the revision, we added rendered frames of the agents (Sec. 5.5, Fig. 8 and App. E.7 Figs 24, 25 and 26), which show how KERL's behavior changes as depth increases. The general answer at the top of the rebuttal lists all the new content.
We also appreciate the reviewer's questions. The question of which change drives KERL's improvement over CRL, the fixed goal encoder or the regression loss, led to a new ablation that separates the two (Sec. 5.6). We believe it substantially strengthens the paper. The questions on high-dimensional goals, on whether Fig. 3 generalises beyond its toy environment, and on why KERL outperforms CRL and SRL also helped us clarify the scope of our claims. We answer each question below, in the order the reviewer asked them.
Q1. The main source of improvement (Fig 9 in the revised paper, depth 16, full tables in our reply to aVMR)
Q2. High-dimensional goals such as images. All our goals are coordinates: Early intuitive response: new Prop. 3.1 does not depend on the goal dimension, which justify the use of random Fourier features for control (but this remain only a theorical result) We fully agree with the reviewer that evaluating Kerl on visual-based settings would further validate its strength, we leave this for futur work and add it in the limitation section.