- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper approaches the field of context-based offline meta-RL, and particularly the problem of learning representations (contexts) that are robust to distributional shifts, either task shifts or representation shifts. It proposes a method that combines three regularizations that works on top of the predictive models supervision and aims to regularize the geometry of the learned embeddings. The paper provides theoretical analysis of these objectives (error bounds) and experiments in MuJoCo and MetaWorld tasks, in several different contexts: generalization/robustness, sensitivity to offline data, and ablations.
Critical strengths
-
The problem of offline meta-RL is very relevant for Deep RL research, as it addresses learning and online adaptation from offline, fixed datasets - this is a big bottleneck while deploying RL-based adaptive systems.
-
The paper presents extensive evidence to support the method, either theoretical and experimental. Empirical results are usually very positive.
Critical weaknesses
-
The main problem of the paper is clarity and method presentation. Overall, it is a really hard to parse paper due to how it is written.
- It is really hard to understand the motivation and contribution of the work from the Introduction. First, it is very much mixed with a literature review that list some works alongside their contributions but not really contextualizes and contrasts them with this work.
- The contribution is described with several technical terms that are so far undefined and failed to provide a clear and intuitive description of the proposed interventions. Terms like “predictive dispersion”, “diversity and stability of local responses”. The names of the proposed interventions are also vague and exhaustively used without a proper clear, intuitive definition: “Adaptive Diversity Control”, “Spectral Functional Diversity”. This makes it really hard to understand and grasp the concepts in the paper and build an intuition of the method, so the scientific communication is really degraded.
- The paper seems to implicitly assume a specific architecture for offline Meta-RL but fails to communicate it for the reader. The paper describes the general offline meta-rl and shifts in the preliminaries, but does not formally describe the predictive models used in the method and assume this is known by the reader while describing the proposed method. It talks about encoders and decoders, predicting supervision for dynamics and reward, but never describes this architecture in the background section. Not all Meta-RL is necessarily model-based, so it is important to describe correctly what is the starting point of the proposed method.
-
I observed that the presented baselines results are considerably worse than the ones reported in prior published work [1], which compromise the claims here. The proposed method still seems to perform better, but the margins are much less wide.
-
I also noticed that the method introduces a lot of hyperparameters, which are also sensitive across tasks. There is no further evidence of robustness of these hyperparameters, nor any description on the protocol followed for their selection. This is important to understand the practical limitations of the proposed method.
I cannot recommend acceptance without a substantial rewrite of its content, particularly Introduction and Methodology.
EDIT POST-Rebuttal: See my comment below.
[1] Li et. al. Towards an Information Theoretic Framework of Context-Based Offline Meta-Reinforcement Learning. NeurIPS, 2024.
Questions for authors
What was the protocol for tuning the hyperparameters? Did you use the same tasks to select them or a held-out validation task set?
3 comments
Dear authors,
Thank you for your efforts on rebuttal. I acknowledge reading your rebuttal and I can see the clarity of the paper improved in the revised draft. Regarding the baselines: I also noticed that the proposed method improves over the baseline published results for specific evaluation setups, but I raise the following concern: this acknowledges that there is a gap between the published results and the ones adopted in this paper, which raise questions whether the other evaluation setups (e.g., OOD, generalization under shifts, etc) are affected due to weaker baselines. In any case, I will assume the best efforts in the experimental setup and acknowledge the fact that sometimes is hard to reproduce results published in papers.
The hyperparameter protocol looks clear to me. It is good to know no tuning specific tuning was needed across tasks, although the sweep suggests the method is quite sensitive to the hyperparameter values.
Overall, I decide to increase my score from 2 to 3.
Dear Reviewer,
Thank you so much for your valubale comments, suggestions and time during the review process. Moreover, we are happy that we could carefully address your comments and are grateful for increase your score from 2 to 3. As you recommended, we will carefully consider addressing your new concern while we are revising the paper.
Thank you for the detailed feedback on presentation and baselines. We have substantially revised the Introduction and Preliminaries, added a dedicated Related Work section, and show below that GEODE's state-of-the-art result holds even against the numbers published in [1].
Motivation and contribution. The Introduction now follows one line of argument: (1) the inferred task representation z is always somewhat wrong, especially under shift; (2) the decoders that train the encoder pass errors through their Jacobian with respect to z (Eq. 3), and reconstruction accuracy says nothing about that Jacobian's shape; (3) GEODE therefore controls three properties of the ensemble directly. The literature now sits in a Related Work section that contrasts each line of work with ours.
Undefined terms. Each term is now defined in Sec. 2, before the method: prediction dispersion (how much the K decoders disagree on the same input), local response (how a prediction changes when z is nudged slightly), its direction (whether decoders change their predictions in different directions; SFD) and its magnitude (how much a prediction changes for a nudge of fixed size; LR). "Spectral" in SFD refers to the eigenvalue spectrum of the response Gram matrix (Sec. 3.3); "latent-space" in LR refers to robustness to errors in z.
Architecture. Sec. 2 now describes the predictive context-based OMRL pipeline. An encoder maps a context to z, and a model-free offline actor-critic acts on z. The decoder ensemble teaches the encoder what z should capture: K decoders predict reward and state change from (s, a, z), and their errors, propagated through each decoder's Jacobian with respect to z (Eq. 3), form the encoder's learning signal. The quality of z therefore depends directly on the ensemble's geometry, which is exactly what GEODE controls, at no extra cost at test time.
Baselines vs. Li et al. (2024). GEODE remains state of the art even against the numbers published in [1]:
Each ID return is divided by the best of the six methods per environment (best ÷ return for HalfCheetah-Vel) and averaged; OOD is excluded because [1] uses a different set of OOD behavior policies. GEODE improves on the best published result on Ant-Dir (+35%) and Reach (+24%), matches it on HalfCheetah-Vel and Walker-Param, and is within 8% on HalfCheetah-Dir and Hopper-Param. No published method is within 20% of the best on every environment; GEODE is.
The margin is largest where task inference is hardest. Trained only on random-policy data (Ant-Dir), GEODE reaches 289 / 291 (ID / OOD), against 81 / 62 for published UNICORN on the same data and even above UNICORN's expert-data result (279 / 262); GEODE keeps 81% / 84% of its expert-data return, against under 30% for published UNICORN and FOCAL. Under task-distribution shift (Fig. 5), GEODE gives the highest zero-shot return in every shifted band of HalfCheetah-Vel, Hopper-Param, and Walker-Param, and remains best or tied after one context episode; in the severe band it reaches 420 one-shot on Walker-Param against 290 for the next best, almost unchanged from its ID level (450).
Where protocols match, our baseline reproduction agrees with [1] (e.g., the data-quality study). Both works build OOD contexts from behavior policies of other tasks, but with different policy sets (ours: 40 source tasks spanning weak, intermediate, and converged behavior), so we compare against [1] on ID returns.
Hyperparameters and tuning protocol. None of GEODE's three loss weights or its perturbation radius was tuned on meta-test performance. Reference values were set once on Ant-Dir from training-time behavior and transferred to every other environment by a fixed rule (App. E.1): radii scale with the latent dimension, the LR weight is rescaled to cancel its dependence on the radius, λ_SFD needs no rescaling because SFD compares normalized responses, and λ_ADC is constant. GEODE thus reaches state-of-the-art performance on six benchmarks with a single reference configuration and no environment-specific tuning. A λ_SFD sweep (Appendix F) shows alignment falling monotonically with λ_SFD (Walker 0.388 → 0.233 → 0.106; Ant-Dir 0.604 → 0.448 → 0.230).
The revised paper is organized around a single argument with every term defined, introduces the architecture explicitly, and establishes GEODE's advantage against the published baselines, with the largest margins where task inference is hardest.
[1] Li et. al. Towards an Information Theoretic Framework of Context-Based Offline Meta-Reinforcement Learning. NeurIPS, 2024.