acceptodds
Under review as a conference paper at ICLR 2027

When Generalization Goes the Wrong Way: The Limits of Zero-Shot Offline Meta-RL

Abstract

Context-based offline meta-reinforcement learning adapts to new tasks by inferring a latent variable while keeping the policy fixed. Can failures beyond the training task distribution be resolved by better task inference, or do they reflect missing behaviors in the policy itself? Across three continuous-control task families and five learners we observe a pronounced asymmetry: magnitude extrapolation often retains useful performance, whereas direction extrapolation can produce returns below a stationary baseline. We analyze this failure by separating latent-selection error from limitations of the frozen policy's behavioral repertoire. In Ant-Dir, the measured heading range over evaluation tasks matches that attained on training tasks in of models; broadening behavior-data coverage at fixed training tasks yields little expansion; and return-based latent search fails to recover three distant goals while recovering two in a full-support control. Together these results support a behavioral-coverage bottleneck that task inference alone does not overcome in any condition we test. Strong in-support returns can meanwhile mask severe behavioral collapse, so we recommend direct behavioral probes alongside return-based evaluation of zero-shot offline meta-RL.

Primary area:
reinforcement learning
Keywords:
offline meta-reinforcement learning, task representation learning, out-of-distribution generalization, test-time adaptation, world models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qz71OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates the limitations of zero-shot generalization in context-based offline meta-RL, focusing on whether OOD failures arise from inaccurate task inference or insufficient behavioral coverage in a frozen policy. Across several continuous-control benchmarks, the authors observe that direction extrapolation is substantially more difficult than magnitude extrapolation. They further introduce a behavioral-coverage perspective and conduct experiments involving training-support variation, behavior-data pooling, latent optimization, and policy adaptation.

Critical strengths
  1. The distinction between task inference and behavioral limitations is an interesting direction for analyzing generalization failures in offline meta-RL.
  2. The paper highlights an important limitation of return-based evaluation: high returns may mask limited behavioral generalization. The behavioral coverage analysis provides a valuable complementary perspective.
  3. The comparison between direction and magnitude extrapolation has valuable insights into how the structure of distribution shifts affects generalization.
Critical weaknesses
  1. The paper has unclear writing. The numerous experiments and analyses are not well connected to a coherent scientific question, making the main contribution difficult to identify. The paper also lacks an intuitive overview figure explaining the problem formulation. Almost every result is immediately followed by extensive qualifications and caveats (more like a patch), which significantly disrupt the logical flow and readability.
  2. The central theoretical observation is largely definitional, and the paper offers limited mechanistic insight into why restricted behavioral repertoires emerge during offline meta-RL training.
  3. The paper does not sufficiently distinguish latent-selection error from limitations of the frozen policy’s behavioral repertoire. The paper does not convincingly distinguish failures in latent selection from limitations of the frozen policy itself. A narrow range of observed behaviors does not necessarily mean that the policy cannot produce other behaviors with different latents. Similarly, the failure of return-based latent search does not establish that successful latents do not exist, especially since the same search procedure fails to recover a task that the full-support model can already solve. Therefore, the experiments do not conclusively establish behavioral coverage as the primary bottleneck.
Questions for authors
  1. The performance gains from policy fine-tuning are presented as evidence that the frozen policy lacks the behaviors required for unseen tasks. However, this comparison does not rule out the possibility that better latent optimization could achieve similar improvements without modifying the policy. This concern is particularly relevant given that the proposed latent search fails even on tasks solvable by the full-support model. Without more rigorous controls, it is difficult to attribute the benefits of policy adaptation specifically to overcoming behavioral-coverage limitations.
  2. The strongest evidence for the behavioral-coverage bottleneck comes from Ant-Dir, where tasks differ only in reward direction and behavioral coverage can be directly characterized by displacement. To what extent do these findings generalize to more complex task distributions involving different dynamics or behaviors that cannot be captured by simple geometric measures?
Reviewer_7GzuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies extrapolation failures in context-based offline meta-RL, distinguishing errors in latent selection from limitations of the frozen policy’s behavioral repertoire. It evaluates five learners across three continuous-control task families, with detailed behavioral analysis in Ant-Dir. Through training-support interventions, pooled behavior data, latent optimization, and conditional performance bounds, it provides evidence consistent with a behavioral-coverage bottleneck. It also shows that strong in-support returns can conceal weak task-dependent behavioral variation.

Critical strengths
  1. The paper identifies a concrete evaluation blind spot. Policies executing nearly the same heading across substantially different requested directions can still achieve strong in-support returns. This provides a useful empirical motivation for complementing return-based evaluation with direct behavioral measurements.

  2. The investigation combines several complementary interventions. Training-support changes, pooled-data experiments, and latent optimization provide more informative evidence than reporting extrapolation returns alone. The paper also clearly acknowledges important limitations of its search procedures and conditional bounds.

Critical weaknesses
  1. The additional conceptual and mechanistic insight is limited. Distinguishing incorrect task inference from insufficient policy capability is a familiar observation; related concerns already appear in MIER. The decomposition formalizes this distinction, but its conceptual novelty appears modest. The pooled-data intervention usefully tests whether broader behavior data is sufficient at fixed training rewards, but does not isolate how representation learning, policy optimization, and reward support contribute to the resulting restriction. Consequently, the paper establishes particular failure patterns and evaluation blind spots more convincingly than a substantially new mechanism.

  2. Finite latent search cannot exclude existing successful behaviors. CEM fails on a full-support target for which the encoder already achieves a normalized return of 1.056, demonstrating that the search can miss known successful latents. Its evaluation also covers only three targets, one search seed per model–goal pair, and one selected pooled-data training seed. Although the authors explicitly acknowledge these limitations, they materially weaken the distinction between behavioral unavailability and difficulty finding an appropriate latent.

  3. The instantiated theoretical bounds do not resolve this uncertainty. The reported bounds depend on an empirical smoothness estimate rather than a proven global bound, and apply only to a restricted latent domain. Moreover, the resulting normalized-return ceilings of approximately 0.73–0.84 remain above the 0.40 failure threshold. These calculations therefore do not independently establish the behavioral unavailability suggested by the unsuccessful searches. This limits their evidential contribution, rather than indicating an error in the conditional theorem.

  4. The empirical scope limits the breadth of the conclusions. All three task families involve locomotion with fixed dynamics and changing rewards or goals, while the detailed behavioral-repertoire analysis primarily concerns Ant-Dir. Direction/magnitude asymmetry is also partly interpretable through these tasks’ reward geometry. Although the authors appropriately qualify their claims, the evidence leaves uncertain which findings extend to structurally different task families.

Questions for authors
  1. What is the strongest new mechanistic conclusion beyond the established distinction between task inference and policy capability, and which experiment most directly establishes it?

  2. Can a stronger or better-calibrated latent search reliably recover the known successful full-support behaviors? Would the restricted and pooled models still fail under that calibrated procedure?

  3. How sensitive are the reported conditional ceilings to the empirical smoothness estimate? Which substantive conclusion remains supported without treating that estimate as a global upper bound?

Reviewer_S9iKOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper focused on context-based offline meta-reinforcement learning and aimed to answer whether better task representations can promote generalization to test distributions. This paper first analyzed the performance gap and decomposed it into two components: the selection gap caused by the policy and the family shortfall caused by the training objective. Then it conducted a series of experiments on MuJoCo benchmark to evaluate how the direction and magnitude of extrapolation affect performance. Finally, it investigated the relationships among action probability, reward prediction error, task representation, and generalization performance.

Critical strengths

It seems that this paper conducted many experiments to evaluate the impact of the training distribution and task representation on test performance. Figure 1 displays the influence of extrapolation magnitude and direction on normalized returns on the MuJoCo benchmark.

Critical weaknesses

In my view, this paper does not propose any new methods. Instead, it focuses primarily on reporting experimental results, with limited presentation of the underlying data and little in-depth analysis of the reasons that may explain the findings. Moreover, the most findings appear largely intuitive. For example, it is unsurprising that adding tasks and trajectories beyond those in the dataset, or optimizing task representations out of the training distribution, may improve generalization. Nearly all experiments, except the one described in Line 438, lack comparisons with baseline methods, making it difficult to assess the effectiveness of this paper.

My primary concern is the overall writing quality of the current draft. This paper uses so many nonprofessional terms related to reinforcement learning, making it difficult to follow and understand. It also contains many doubtful or improper expressions such as "full-suport control," "in-support returns,"behavioral repertoire"," "latent search," and so on. Besides, many notations in the main text are undefined, includings Jσ(u)J_\sigma(\mathbf{u}), dneard_{near}, h^p(u)\hat{h}_{p}(\mathbf{u}), and so on. The poor writing quality of this paper significantly impedes understanding of the paper.

Questions for authors
  1. Is ψ(⋅)\psi(\cdot) implemented using neural networks? How is ψ(⋅)\psi(\cdot) trained?

  2. What does Jσ(u)J_\sigma(\mathbf{u}) denote in Eq.(1)? What specific rules, represented by σ\sigma, are applied in the experiment?

  3. What does dneard_{near} denote, and why is the Spearman correlation computed in Line 188??

  4. What do Q-F (Line 207) and C-F (Line 208) stand for, and why was this metric chosen?

  5. How is A^θ\hat{A}_ \theta implemented in Line 314? What does h^p(u)\hat{h}_ {p}(\mathbf{u}) denote, and why is the bound introduced in Line 422?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.