acceptodds
Under review as a conference paper at ICLR 2027

Trust, but Verify: Test-Time Task Inference in Offline Meta-Reinforcement Learning

Abstract

Context-based offline meta-reinforcement learning adapts from a short history of transitions: an encoder turns the history into a task code, and a policy acts conditioned on that code. Outside the training range, return can fall for several reasons: the policy may be unable to solve the new task, the inferred code may be wrong, or the task may lie too far from anything seen in training. We derive an exact four-term decomposition of this lost return and show how to estimate its terms from checkpoint rollouts and data-collecting experts. The measurements reveal two failure modes. On manipulation tasks the frozen policy already solves the new task when given a suitable training code, so the choice of code fails; on locomotion tasks with shifted rewards the oracle-context code is already near the best available and leaves little code-selection headroom, while self-collected context can lose further return. Either way, choosing a code at test time is unreliable: the encoder can infer one from exploration episodes, and a decoder can select the training code that best explains them, but each wins where the other loses and no pre-deployment statistic reliably identifies which one to trust. We propose VeriCode, which explores with a spread of training codes and immediately keeps one seen to succeed. If none succeeds, it compares an encoder candidate and a likelihood candidate through one-episode verification, except when both heads identify the same training code; without a success signal, it compares candidates by return. Under independent evaluation episodes, the probability of selecting the worse of two fixed candidates decays exponentially with the verification budget. Across six Meta-World and six MuJoCo families and three algorithms, VeriCode is at or above the strongest comparator in every cell, raising Meta-World success from 0.52 to 0.83 and MuJoCo return from 200 to 267, at mean adaptation budgets of 5.5 and 9.4 episodes, respectively.

Primary area:
reinforcement learning
Keywords:
offline meta-reinforcement learning, test-time adaptation, task inference, out-of-distribution generalization, task representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_z3WwOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies context-based offline meta-reinforcement learning and characterizes the loss when test tasks fall outside the training distribution. The authors further propose VeriCode, which demonstrates empirical success across six Meta-World task families, six MuJoCo task families, and three algorithms. However, the paper’s unclear presentation makes it difficult to assess its technical contributions.

Critical strengths

The paper works on an important problem of test-time task inference under distribution shift in context-based offline meta-reinforcement learning, and it provides a theoretical characterization of performance loss and the proposed VeriCode method.

Critical weaknesses

The presentation and logical flow are difficult to follow. Several terms, assumptions, and methodological details need clearer explanations. For example:

  1. The introduction uses ambiguous words and introduces concepts without sufficient context. In the third paragraph, the authors state that “the two benchmarks turn out to fail for different reasons.” Benchmarks themselves do not fail; the authors should identify which algorithms fail on these benchmarks. Similarly, statements such as “On cheetah-vel, extrapolation dominates” and “loss is contextual” need precise explanations. What is being extrapolated, and what does “contextual” mean here? “Regime C” is also mentioned without being introduced. Finally, the statement “This points at procedures that change nothing about training and act only on which code is deployed” is unclear: "This points at procedures" refers to which points?

  2. Terminology should be defined when first introduced. For example, “OOD” appears in the introduction without a clear definition. And later in section 3, the authors mention ""OOD" describes the experiments".

  3. In section 3, Expressions such as “we make precise the return a deployed context-conditioned policy leaves on the table in an OOD task” are hard to read. K\mathcal{K} is said to be a reachable set, but the definition of a reachable set is missing. Equation (2) appears to follow from a standard finite-horizon performance difference lemma. The authors should cite the relevant literature and clarify which part of the theorem is new. And in Theorem 1, it suddenly mentions policy-sensitivity assumption, action-Lipschitz assumption, but these assumptions are not introduced in the main paper and not referred to the appendix position.

Questions for authors

The paper should provide pseudocode for vericode specifying its inputs, verification steps, code-selection procedure, and output. How does the method apply to general tasks beyond its implementation in Meta-World?

Reviewer_hKwmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work studies test-time task inference for offline meta-reinforcement learning. Prior methods such as IDAQ rely on model confidence scores (encoder embedding distance or dynamics likelihood) to select task encodings from a fixed codebook, assuming that model uncertainty reliably reflects the true quality of candidate encodings. However, the authors empirically show this assumption often fails: observations can be indistinguishable across different task encodings even though their resulting policies perform drastically differently, a phenomenon they term observational unidentifiability. To diagnose the sources of OOD performance drop, they design a return decomposition to separate losses from policy extrapolation, context distribution shift, and task-encoding inference mistakes. Motivated by this analysis, they propose VeriCode, a lightweight test-time wrapper that leverages limited online rollouts to directly verify candidate encodings instead of trusting model confidence. VeriCode maintains encoder and likelihood candidates; when candidates conflict, it validates them via environment rollouts. It also uses FPS-guided exploration with early termination upon successful trials. Evaluated on MetaWorld and MuJoCo with multiple offline meta-RL backbones (CSRO, UNICORN, ER-TRL), VeriCode outperforms IDAQ under matched test-time interaction budgets by better utilizing the return/success signal from online episodes.

Critical strengths
  1. It presents a sensible diagnostic decomposition that breaks OOD return degradation into four interpretable components, which disentangles failure sources of offline meta-RL task inference.
  2. It provides solid theoretical derivations, formalizing observational unidentifiability and establishing propositions to support the design of VeriCode.
  3. Extensive experiments validate the method across two environments and three backbones. Careful baselines (IDAQ5/IDAQ7) with matched interaction budgets confirm the gains come from better use of environmental feedback.
Critical weaknesses
  1. The method essentially selects a pre-existing training context closest to the test-time situation. It abandons the potential of context-based meta-RL backbones to generate superior contexts. While this limitation holds under current models, future stronger backbones may unlock better context construction.
  2. The evaluation setup raises fairness concerns across both MetaWorld and MuJoCo. VeriCode leverages return or success signals from exploratory rollouts to select the best task encoding, and its performance gain is especially pronounced on MetaWorld, where binary success signals enable straightforward identification of promising candidates from exploratory rollouts. In contrast, baseline methods such as IDAQ only extract trajectory observations from these rollouts while discarding reward and success information. Baselines could also achieve better performance if they were permitted to utilize rollout returns or run multiple trials and select the maximum return.
  3. The method may fail in stochastic environments.
Questions for authors

Do resets in the test environment always return the same state?

Reviewer_zmFCOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper analyzes several factors that contribute to declining returns in context-based offline meta-reinforcement learning (meta-RL), focusing on the policy, task representations, and task distributions. The paper then decomposes the policy performance gap along these dimensions. The authors argue that selecting task representations during meta-testing can be unreliable. To address this issue, they propose VeriCode, which selects more accurate task representations based on returns or downstream task success. Experiments on MuJoCo and Meta-World demonstrate the effectiveness of the proposed method.

Critical strengths

The experimental results on Meta-World in Table 1 and Figure 2 show that VeriCode outperforms IDAQ methods, even on out-of-distribution (OOD) tasks.

Critical weaknesses

This paper does not provide sufficient motivation for why task representation is unreliable during meta-testing, and the proposed method is not novel and is more likely to be a data-filtering approach. Moreover, the logic of the whole paper is messy. The paper discusses experimental results as early as the Introduction, but reports the specific values only in the text and provides little in-depth analysis of the reasons behind them. The same issue arises in Section 4. Although the paper presents numerous theorems, the proofs in the Appendix are lengthy and difficult to follow. Some proofs do not appear to establish their stated conclusions, as in Lemma 1, while others rely on assumptions at critical steps in Line 1157, as in Theorem 2-A. Although there are many offline meta-RL baselines, this paper compares the proposed method only with IDAQ, which was published in 2023, making it difficult to comprehensively evaluate the method's competitiveness.

My main concern also lies in the overall very poor quality of the writing in the current draft. The frequent use of nonprofessional reinforcement learning terminology makes the paper hard to follow, such as "do not say why," "read the improvement off the OOD table," "source-task telescoping structure," and so on, further undermining its clarity. Moreover, in meta-RL, the inferred task ID by the context encoder is generally termed as task representation instead of task code.

Questions for authors
  1. How does the decoder select the training task representation, and how is it scored?

  2. Does this paper consider both online and offline settings or does "online" in Line 109 refer to online testing phase but still in an offline setup?

  3. What do m(s,z)m(s,z) in Line 191 and fif_i in Line 293 denote? Many symbols in the paper are not defined when they first appear.

  4. What conclusion can be drawn from the bound in Theorem2-A?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.