- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper asks whether better goal representations improve offline GCRL. In OGBench mazes, temporal distances can be computed exactly with BFS, so the authors build an exact (ideal) dual goal representation. They then corrupt it step by step, while keeping the downstream learner (GCIVL, CRL) fixed. Performance barely changes across these variants, and even a random goal code performs about the same. Applying the same codes to the agent's current state instead raises success substantially. From this, the authors conclude that the state pathway is the bottleneck. They then propose appending random Fourier features of the agent's x/y position to the observation. This raises GCIVL success on antmaze-large from 32 to 84 and roughly doubles it on humanoidmaze-medium.
Critical strengths
- The goal-side analysis is a useful negative result for the GCRL representation literature.
- The limitations and implications of the paper are clearly and honestly stated.
Critical weaknesses
-
The title and framing don't match the content. The paper is titled and motivated around goal representations, but its main constructive contribution (position features) is on the state side. The two halves are connected by Section 5.1, but it reads more like "state inputs matter more than goal representations".
-
The goal representation results (Table 1) are slightly problematic, since several CIs span approx. +- 10 points (and most CIs include zero) and some environments use only 2 seeds. The authors acknowledge that CIs exceed their 5-point tolerance, but "no effect" currently can't be distinguished from moderate effects.
-
The authors correctly present the random Fourier features as a standard construction. But presenting it as "our proposal" overstates the contribution. Prior RL work that applies Fourier features to a closely related problem is missing (Konidaris et al., 2011; Yang et al., 2022). Further, the method also needs to know which observation dimensions are x and y positions, so it has the notion of feature engineering. Combined with the narrow evaluation (two environments, GCIVL only), the constructive contribution is limited. The paper therefore rests mainly on the goal-side analysis, which is useful but itself narrow in scope.
-
As the authors themselves state, the method/gains don't transfer to manipulation or pointmaze. That makes the method a maze-specific fix. My impression so far (as in Section 5.2): neural nets struggle with low-dimensional coordinate inputs, lifting them into a high-dimensional code helps, and the distance geometry inside that representation barely matters.
References:
- Konidaris, George, Sarah Osentoski, and Philip Thomas. "Value function approximation in reinforcement learning using the Fourier basis." Proceedings of the AAAI conference on artificial intelligence. Vol. 25. No. 1. 2011.
- Yang, Ge, Anurag Ajay, and Pulkit Agrawal. "Overcoming the spectral bias of neural value approximation." arXiv preprint arXiv:2206.04672 (2022).
Questions for authors
- Does goal-representation quality start to matter once position features are added? If the state pathway was limiting the policy, the corruption ladder from Figure 4 might no longer be flat with position features in place.
2 comments
While I do appreciate the authors for their additional experiments and effort, I sincerely think the main results are too noisy, and the paper lacks informativeness. Sadly, I learned little.
-
W2: Is not adressed. I think, your main figure is Table 1 and the results are what they are. Negative results are fine, but the informativeness is limited unfortunately. Then using positional features + further analysis is cool and it improves performance. But this was previously known, and is not a significant contribution here ...
-
W4: Based on the experiment, feature engineering (selecting only x,y) is actually important. Otherwise performance degrades.
Best regards.
Dear Reviewer,
Thank you for your efforts in reviewing this submission. As the discussion period is approaching its end, I would like to kindly remind you to read the authors' rebuttal and engage in the discussion. Your input is important for reaching a well-informed final assessment.
Thank you for your time and cooperation.