acceptodds
Under review as a conference paper at ICLR 2027

Do Better Goal Representations Improve Goal-Conditioned Reinforcement Learning?

15 views

Abstract

Goal-conditioned reinforcement learning (GCRL) relies heavily on how target goals are represented to the policy. While recent methods encode goals via temporal distance, occupancy, or controllability, it remains unclear how much downstream performance actually depends on representation quality. We study this in offline GCRL by constructing an exact temporal-distance goal representation in deterministic mazes. We then systematically corrupt its geometric quality while keeping the downstream learner fixed. Across OGBench navigation tasks and two algorithms, large changes in goal-representation quality produce almost no change in performance. However, applying the same interventions to the agent's current state more than doubles success, revealing the state pathway as the true bottleneck. Building on this insight, we show that simple random Fourier positional encodings substantially improve performance on the hardest navigation tasks without map information or objective modifications. Overall, our findings suggest that in state-based offline navigation, improving how the agent's current state is represented matters far more than refining the goal representation.

Primary area:
reinforcement learning
Keywords:
Goal-Conditioned Reinforcement Learning, Offline Reinforcement Learning, Goal Representations, State Representations, Representation Learning, Positional Encoding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MTmjOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether better goal representations improve offline GCRL. In OGBench mazes, temporal distances can be computed exactly with BFS, so the authors build an exact (ideal) dual goal representation. They then corrupt it step by step, while keeping the downstream learner (GCIVL, CRL) fixed. Performance barely changes across these variants, and even a random goal code performs about the same. Applying the same codes to the agent's current state instead raises success substantially. From this, the authors conclude that the state pathway is the bottleneck. They then propose appending random Fourier features of the agent's x/y position to the observation. This raises GCIVL success on antmaze-large from 32 to 84 and roughly doubles it on humanoidmaze-medium.

Critical strengths
  • The goal-side analysis is a useful negative result for the GCRL representation literature.
  • The limitations and implications of the paper are clearly and honestly stated.
Critical weaknesses
  • The title and framing don't match the content. The paper is titled and motivated around goal representations, but its main constructive contribution (position features) is on the state side. The two halves are connected by Section 5.1, but it reads more like "state inputs matter more than goal representations".

  • The goal representation results (Table 1) are slightly problematic, since several CIs span approx. +- 10 points (and most CIs include zero) and some environments use only 2 seeds. The authors acknowledge that CIs exceed their 5-point tolerance, but "no effect" currently can't be distinguished from moderate effects.

  • The authors correctly present the random Fourier features as a standard construction. But presenting it as "our proposal" overstates the contribution. Prior RL work that applies Fourier features to a closely related problem is missing (Konidaris et al., 2011; Yang et al., 2022). Further, the method also needs to know which observation dimensions are x and y positions, so it has the notion of feature engineering. Combined with the narrow evaluation (two environments, GCIVL only), the constructive contribution is limited. The paper therefore rests mainly on the goal-side analysis, which is useful but itself narrow in scope.

  • As the authors themselves state, the method/gains don't transfer to manipulation or pointmaze. That makes the method a maze-specific fix. My impression so far (as in Section 5.2): neural nets struggle with low-dimensional coordinate inputs, lifting them into a high-dimensional code helps, and the distance geometry inside that representation barely matters.

References:

  • Konidaris, George, Sarah Osentoski, and Philip Thomas. "Value function approximation in reinforcement learning using the Fourier basis." Proceedings of the AAAI conference on artificial intelligence. Vol. 25. No. 1. 2011.
  • Yang, Ge, Anurag Ajay, and Pulkit Agrawal. "Overcoming the spectral bias of neural value approximation." arXiv preprint arXiv:2206.04672 (2022).
Questions for authors
  • Does goal-representation quality start to matter once position features are added? If the state pathway was limiting the policy, the corruption ladder from Figure 4 might no longer be flat with position features in place.
2 comments

Dear Reviewer,

Thank you for your efforts in reviewing this submission. As the discussion period is approaching its end, I would like to kindly remind you to read the authors' rebuttal and engage in the discussion. Your input is important for reaching a well-informed final assessment.

Thank you for your time and cooperation.

While I do appreciate the authors for their additional experiments and effort, I sincerely think the main results are too noisy, and the paper lacks informativeness. Sadly, I learned little.

  • W2: Is not adressed. I think, your main figure is Table 1 and the results are what they are. Negative results are fine, but the informativeness is limited unfortunately. Then using positional features + further analysis is cool and it improves performance. But this was previously known, and is not a significant contribution here ...

  • W4: Based on the experiment, feature engineering (selecting only x,y) is actually important. Otherwise performance degrades.

Best regards.

Reviewer_LVW2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether the quality of a goal representation limits performance in offline goal-conditioned RL. They define the "ideal" representation using an exact temporal distance that Dual Goal Representations (Park et al., 2026) approximates. This representation is fed into two methods (GCIVL and CRL) and compared to the learnt DGR representation. They also evaluate a corrupted version of this representation (using noise). Across five navigation tasks the exact, corrupted and random goal representations are statistically indistinguishable from the learnt one. Applying the same per-cell code to the agent’s state instead of the goal raises GCIVL on antmaze-large from about 30 to about 70, and a map-free alternative, random Fourier features of the agent’s coordinates appended to the observation, raises it from 32 to 84 and roughly doubles success on humanoidmaze-medium. Ablations rule out actor capacity, input scale and goal-input width as explanations. Appendices additionally report an interface sweep in which cross-attention over landmark tokens outperforms concatenation, and show that none of the interventions help on pointmaze or on manipulation tasks. The claimed contributions are the measurement of an exactly computed "ideal" goal representation, the identification of the state pathway as the bottleneck, and the position-feature method.

Critical strengths

The paper clearly introduces an important unsolved question and the experimental design is well suited to answer it: the same intervention is applied to the goal and to the state with the learner fixed, null results are acknowledged rather than hidden, and Welch intervals are mostly reported. This matters because the method of the paper is sound even where I disagree with its conclusions; the 40-point gap in figure 5a and the gains on antmaze are real results.

Critical weaknesses
  1. The second contribution is overstated and contradicted by the paper’s own data. It is supported only by antmaze-large, and table 6 (in the appendix...) shows that a more powerful interface does take advantage of a better goal representation: +33 points with the exact representation against +4 with the learnt one. The correct conclusion is that concatenation cannot exploit goal quality, not that goal quality does not matter. The abstract, the title and the first two contributions rest on the stronger reading, so this is the main reason I cannot recommend acceptance as it stands. The first contribution has the same problem in a smaller way: the representation is called "ideal" on the basis of a sufficiency result (line 130) that is not extended to the truncated eq 1, and being sufficient is not being ideal.

  2. The evaluation is too narrow and inconsistent for confident conclusions. The number of seeds looks random (2, 4, 5 or 8) and is never justified; 2 seeds is not enough for a null result, and the one setting where corruption does hurt (pointmaze-large, CRL) is dismissed on variance with the same seed count. Most results are on antmaze only. The third contribution, the paper’s only positive result, is shown on two tasks with GCIVL alone: antmaze-medium and CRL are omitted without explanation, the wavelength sweep stops at its best value, and "roughly doubles" has no table, seed count or interval. Standardising two coordinates making the agent four times worse (32.2 to 7.1) is not explained and looks like a bug. Null results need more power than positive ones, not less. As it stands the paper's conclusions about state-based offline navigation are conclusions about one maze family.

Questions for authors
  1. Table 6 shows that cross-attention extracts +33 points from the exact goal representation but only +4 from the learnt one. How do you combine that with the claim that goal representations are saturated? Is the correct statement not that concatenation cannot exploit goal quality, rather than that goal quality does not matter?
  2. Standardising the two coordinate dimensions drops GCIVL on antmaze-large from 32.2 to 7.1, in a network with layer normalisation. Can you rule out an implementation error, and if so, what mechanism explains a four-fold collapse from a linear rescaling of two inputs?
  3. Why are the corrupted arms in table 3 run with two seeds, how were seed counts chosen across experiments (2, 4, 5 or 8), and on what basis is pointmaze-large under CRL set aside as variance when the same seed count supports the null results elsewhere?
  4. What are the seed counts and intervals for figure 6, and why were CRL and antmaze-medium not run with position features?
  5. Table 6, Figure 11, Table 3 and Appendix J all bear directly on the main claim. Which of these would you bring into the main text, and would the abstract and contributions be reworded to match?
3 comments

Dear Reviewer,

We have uploaded a revised manuscript PDF (see blue colour) that incorporates your feedback. We would appreciate it if you could review the revised manuscript and our responses above. Please let us know if you have any further questions. Thank you once again for your careful review and constructive comments.

Best wishes,

Authors

Dear Reviewer,

Thank you for your efforts in reviewing this submission. As the discussion period is approaching its end, I would like to kindly remind you to read the authors' rebuttal and engage in the discussion. Your input is important for reaching a well-informed final assessment.

Thank you for your time and cooperation.

I would like to thank the authors for this thorough rebuttal. For clarity, I will keep your numbering in this answer. However, this numbering does not correspond to the one I used. Some of my concerns are resolved.

Among the concerns you did not concede, the following are not resolved:

W3/Q3.

I understand that you are working with limited compute but you are not answering my questions. The compute limit justifies using fewer seeds, not different number of seeds across evaluations. Why didn't you use the same number of seeds everywhere? There is probably bias introduced by the number of seeds choice. This is amplified by the fact that at some point you consider a two seed result and at another you dismiss it (as mentioned in my original review).

I would complain significantly less if you used 4 seeds everywhere than with your mix of seed numbers. I have never seen that before.

Whilst I agree that the 28 points difference in your example looks important, 2 seeds is few enough to arise by sheer luck. RL is known for being unstable and often collapsing. Two seed results are fine for intuition but not for a paper.

On that side, your confidence intervals do not solve that problem because RL results do not verify the assumptions (e.g. normality) and such a low number of seeds means that your SD is too stochastic to be used anyway.

The following point remains: "The evaluation is too narrow and inconsistent for confident conclusions. The number of seeds looks random (2, 4, 5 or 8) and is never justified; 2 seeds is not enough for a null result, and the one setting where corruption does hurt (pointmaze-large, CRL) is dismissed on variance with the same seed count."

W5 / Q2

Thank you for these additional experiments. I agree with your conclusion but it shows something important. Do you think that a similar scaling effect could have affected your other experiments? The table suggests so considering that scaling has more impact than learnt vs exact.

An interesting result (that would not move my rating) would be to check if the other results are mostly due to magnitude.

Conclusion

The question the authors are trying to solve is interesting. However, I do not think the paper provides enough to answer it. The experimental setup needs to be overhauled and the framing of the paper needs to be changed too much for a rebuttal.

I am not increasing my rating.

Reviewer_n5LcOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

Summary

The paper asks whether better goal representations actually improve downstream control in offline goal-conditioned RL. Using Dual Goal Representations (DGR), where a goal is represented by its temporal distances to a set of landmark states, φ(g)=(d(a1,g),…,d(aK,g))\varphi(g) = (d(a_1, g), \dots, d(a_K, g)), the authors construct an exact ("ideal") representation in deterministic OGBench mazes with breadth-first search. They then vary its quality while keeping the downstream learner (GCIVL or CRL) fixed:

  • Quality measure. Quality is the Spearman correlation between min-plus decoded distances, d^(s,g)=min⁡i[d(ai,s)+φi(g)]\hat d(s, g) = \min_i [d(a_i, s) + \varphi_i(g)], and true geodesic distances.
  • Corruptions. The exact representation is corrupted with Gaussian or far-field noise, and also replaced by a purely random N(0,1)\mathcal{N}(0, 1) code per goal cell.

They report that the exact, corrupted, learned, and random goal representations all yield statistically indistinguishable success rates. Applying the same random code to the agent's state instead raises success on antmaze-large by about 40 points. From this they conclude that the state pathway, not the goal representation, is the bottleneck. They propose appending fixed random Fourier features of the agent's xyxy coordinates to the observation, which raises GCIVL on antmaze-large from 32% to 84% and roughly doubles success on humanoidmaze-medium. Control experiments rule out network capacity, input scale, goal-input width, and shortest-path routing inside cross-attention as explanations.

Critical strengths

Pros

  1. The experimental design is carefully controlled. The paper constructs an exactly computable ideal goal representation and corrupts it in a controlled way, changing only the representation while the downstream algorithm, objective, and hyperparameters stay fixed. It uses matched interventions that apply the identical code to the goal side and the state side. It measures representation quality with a probe-free min-plus decoding metric, motivated by the observation that learned probes saturate in mazes. Results are reported with Welch confidence intervals. This makes it a useful empirical sanity check on a common premise of the goal-representation literature.
Critical weaknesses

Cons

  1. There is no experimental or theoretical explanation of why the goal-side representation does not matter while the state-side representation does.

    • The paper shows the phenomenon but not its cause. It never explains why the downstream learner ignores the reachability structure in the goal representation and uses little more than goal identity, nor why a distinguishable encoding of the agent's own position helps so much.
    • The authors acknowledge the mechanism remains open (Section 7). Their own frequency sweep contradicts the standard spectral-bias explanation for Fourier features: lower frequencies work best (App. F.2).
    • The ruled-out confounds (capacity, input scale, goal width, attention routing) say what the effect is not, but offer no positive account. There is no theoretical argument connecting state encoding to value or policy learning in goal-conditioned RL.
    • The two downstream learners are flat methods (GCIVL, CRL) that reach only about 30% on antmaze-large. The "state bottleneck" may be specific to these weak learners rather than a general property of goal-conditioned RL; hierarchical methods reach about 90% on antmaze-large-navigate.
    • Without such an explanation, the central claim reads as an empirical observation on a narrow setting rather than an insight.
  2. The claim is too general for the evidence, which is limited to maze navigation.

    • The main experiments use only deterministic state-based mazes. There, the agent's xyxy position nearly determines the task-relevant state, temporal distance equals a grid geodesic, and distances are comparatively easy to capture. This is exactly why the ideal representation is computable there, and also why a positional encoding of xyxy can help.
    • In tasks such as robotic manipulation, there are many possible paths to a goal and no single pair of coordinates locates the agent. The reachability structure encoded in the goal representation may matter much more in such settings.
    • The paper's own appendix (App. J) shows that the interventions do not transfer to manipulation (cube-single-play, scene-play) or to pointmaze, and pixel-based domains are untested.
    • Despite this, the title and framing pose and answer a general question about goal representations in goal-conditioned RL. The conclusions should be restricted to state-based maze navigation, or supported with experiments where the goal structure is not trivially geometric.
  3. The null results are not statistically established.

    • The central claim, that goal-representation quality does not matter, rests on confidence intervals that include zero, using only 2–8 seeds per arm (e.g., 2 seeds for several settings in Table 3 and for CRL on antmaze-medium in Table 1).
    • A non-significant difference is not evidence of no difference. Several intervals span ±10 points or more, and the authors concede that their own 5-point tolerance is not met in all cases.
    • Equivalence tests (e.g., TOST) or substantially more seeds are needed to support the claim that goal representations are "saturated".
  4. The novelty of the proposed position features is limited.

    • Random Fourier features of input coordinates are a standard technique (Rahimi & Recht, 2007; Tancik et al., 2020), and have been studied for value and policy networks in RL before, e.g., Yang et al., "Overcoming the Spectral Bias of Neural Value Approximation" (ICLR 2022). That RL work is not discussed.
    • The method also requires knowing which observation dimensions correspond to the agent's xyxy position, and its best frequency is tuned in maze-cell units. The claim that it uses "no privileged information" is therefore somewhat overstated.
Questions for authors

Already mentioned in the weaknesses section

1 comment

Dear Reviewer,

Thank you for your efforts in reviewing this submission. As the discussion period is approaching its end, I would like to kindly remind you to read the authors' rebuttal and engage in the discussion. Your input is important for reaching a well-informed final assessment.

Thank you for your time and cooperation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.