acceptodds
Under review as a conference paper at ICLR 2027

What In-Context Reinforcement Learners Infer About Unseen Tasks

Abstract

In-context reinforcement learning agents adapt to new tasks from a prompt alone, and are widely reported to generalize to unseen tasks. We ask where an agent's implicit task estimate goes when the test task lies outside the convex hull of the pretraining task parameters. We build region-disjoint parameter splits over locomotion and manipulation families and decode that estimate three independent ways: linear probes on rollout-time hidden states, inversion of emitted actions against per-task expert policies, and direct behavioral readout. All three agree that the estimate saturates at the hull boundary rather than failing erratically: an agent given a task beyond its training range acts as though it had been given the nearest task inside that range. The confinement is not a data limitation, since model-free audits show that the context identifies the out-of-hull task exactly, and it is not specific to attention, since classical context-encoder agents pin their task codes at the same boundary. Correcting the estimate does not repair the failure and can invert it: supplying the true out-of-hull task code degrades return below that of the boundary-pinned agent, because the policy is competent only at conditioning codes it saw in training. Nor is the boundary an artifact of coverage: retraining on exactly relabeled tasks that extend the hull moves the saturation point to the new boundary and no further. Hull confinement thus bounds what these agents infer, and repairing the inference alone does not restore what they can do.

Primary area:
reinforcement learning
Keywords:
in-context reinforcement learning, offline meta-reinforcement learning, out-of-distribution generalization, task inference, task representation learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4VdaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the behavior of in-context RL when exposed to held-out tasks from outside the convex hull of training support. They elect to estimate the agent’s implicit task belief and find saturation outside the training “band.” They use decision-transformer (and its variants) along with some meta-RL context encoders. This saturation is hypothesized to be caused by the authors' proposed "convex aggregation". Using an extra hand-tuned statistic channel, they find mixed results in success, depending on whether it’s ID or OOD, but find better extrapolation.

Critical strengths
  • I can appreciate the selection of the problem here. I do believe isolating a model’s latent inference of a task could be a good analytical tool or pathway for understanding in-context generalization.
Critical weaknesses
  • A critical weakness of this paper is that there is an unclear contribution with a poorly written delivery. The introduction reads like a summary without a clear description of why one should care about the topic and how it relates to broader impact. Some parts of the paper are nearly unreadable such as the Discussion section. The experimental section itself is filled with numerical details and sparse takeaways and insights for the community.
  • Another weakness is the scope of the paper which may be too narrow for a conference main track submission. This further cements how the submission is incomplete and not fit for acceptance. From what I can understand from the discussion: “we recommend also reporting the parameter region covered in training and each evaluation task’s position relative to it.” The paper as it stands could perhaps be condensed into a couple pages or so, and a strong method for overcoming the issues of generalization of task inference could be studied in detail.
  • Lastly, the results of this paper don’t seem particularly conclusive. The paper primarily discusses HalfCheetah-Vel in the results, with no solid reporting of other tasks (e.g., in Figures 1 and 2). Many things are also marked as exploratory.

Writing nitpicks:

  • In the abstract, “requested outward change” is unspecified. Generally, the abstract should contain higher-level details, but this one seems to use jargon which would be elucidated only by reading the paper itself. Reading onwards, it seems like it’s properly covered around like 147-9.
  • The first paragraph reads like a related works section. It’d be much nicer to be introduced to the motivation and issues at hand from a bird’s eye view and get a hint at your key insights.
  • What does line 50 mean, I am not sure what “contaminated” refers to.
  • Continuing the read the intro, I feel strongly that it is not calibrated as an intro but seemingly a kind of “rapid summary” of the paper itself. - I’d appreciate if the introduction was more clear about the gravity of this problem, the issues that the current style of thinking has caused, the desiderata, and the core insight you’d like to share with the community.
  • By Section 3, I’m still a bit confused on what the work’s contributions are and what I’m in for when I’m reading. For the most part, there seem to be very “appendix-like” details embedded in the first glance of what your method/contribution’s section will show.
  • Not sure why on lins 209-15 there’s a dump of numerical data; it’s a bit hard to parse and probably doesn’t belong in text. To be honest, a majority of page 5 is like this too. Perhaps using the text to communicate the main message and keeping visualizations to communicate numbers would be a better policy
  • The results continue to read quite overloaded with jargon, numerical details, and very low-level observations.
  • The discussion is nearly incomprehensible with very unclear contributions.
Questions for authors

None

Reviewer_X5g2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks what in-context RL agents infer when the test task is outside the convex hull of the training tasks. The authors train Prompt-DT and a small DPT on 14 MuJoCo and MetaWorld task families, and read the agent's task estimate with linear probes, action inversion against per-task experts, and the agent's behavior. On HalfCheetah-Vel the estimate stops near the training boundary although the prompt contains the right target, and three classical context encoders show the same pattern. The paper also tests a hand-designed context statistic, an oracle task code and a wider training range, and proves two theorems about convex and affine readouts.

Critical strengths
  1. The question is well motivated. Most ICRL papers evaluate on held-out tasks inside the training range, which only tests interpolation. This paper makes that point clearly and builds test tasks outside the hull.

  2. The HalfCheetah-Vel result is clean. The agent stops near the edge of the training range, while the SAC expert can still solve those tasks and the prompt carries the target. The classical encoders fail in the same way, so the finding is not specific to one architecture.

Critical weaknesses
  1. The paper does not really measure what the agent infers. All three readouts are taken while the agent is acting, so they all follow what the agent does. A probe trained on in-range rollouts can simply read the current speed, and the layer-0 control shows the input alone already gives most of it. So "the agent inferred the wrong task" and "the agent inferred the right task but cannot run that fast" look the same here, and the paper's own oracle experiment shows the second case does happen. This is the question in the title, and for Prompt-DT it is left open.

  2. The evidence is too thin for the claims. Only HalfCheetah-Vel gives a clean reading. The other 13 families either fail the decoder check or only show behavior. The intervention needs hand-designed statistics, its families were selected using the test bands, and it hurts in-range performance. The two theorems depend on constants that are never measured and on assumptions the experiments violate, so they do not explain the results. In the end this is one case study presented as a general finding.

Minor issues:

  • The Ant-Goal extreme tasks are built with np.linspace without a num argument, which gives 50 points, and only the first 10 are used. The "extreme" band is therefore a small arc right next to the moderate band, which is also why tasks 29 and 30 are duplicates.
  • Some statements do not match the tables or the code. For example, the same in-hull slope is attributed to three different instruments in different places, and the paper says both new arms add about 26k parameters while statsC adds about 7k.
  • The supplementary code leaves out the probe, the action inversion and mini-DPT, so the main measurements cannot be checked.
  • The paper is hard to read. Almost every sentence is hedged, and some hyphened terms are never defined, e.g. "kill-test verdict".
Questions for authors

See Weaknesses. Also,

  1. Can you read the task estimate with the states held fixed and only the prompt changed? For example, probe the first step, where MetaWorld observations are identical across tasks, or replay one fixed trajectory under different prompts. If the estimate still stops at the edge, I would be happy to raise my score.
  2. Which return-to-go target is used on held-out tasks? The code reads it per task from meta.json. If it is the expert return of that task, the return token leaks task information outside the prompt.
  3. Does the saturation hold with more training tasks or a larger model? With 20 tasks and a 3-layer model, the agent may simply be retrieving the closest training task.
Reviewer_9u7dOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates whether ICRL agents can infer tasks outside their training distributions and whether their failures arise from task inference or policy execution. The paper uses task-parameter readouts, behavioral analysis, and controlled interventions to study this distinction. The results suggest that agents often fail to extrapolate beyond the training task range, and that providing more accurate task information does not necessarily lead to better behavior.

Critical strengths
  1. The paper investigates what ICRL agents infer about unseen tasks, going beyond performance-based evaluation to distinguish task inference from policy execution. This is an important and underexplored question with clear research value.
  2. The authors investigate OOD generalization through complementary methods, including task readouts, behavioral analysis, and controlled interventions across diverse tasks. This provides a comprehensive perspective on the potential sources of generalization failure.
  3. The paper complements its empirical findings with theoretical results that explain potential limitations of task extrapolation and analyze alternative task-readout mechanisms, providing a principled foundation for understanding OOD generalization.
Critical weaknesses

The main concern is that the paper draws a broad conclusion from Prompt-DT and mini-DPT, two methods introduced at least three years ago and trained and evaluated on a limited range of task families.

  1. Tasks such as HalfCheetah-Vel and Ant-Goal represent a narrow form of adaptation involving hidden instructions or goals. ICRL encompasses a much broader range of adaptation settings, so it is unclear how far the findings generalize.
  2. Could the breadth and diversity of training tasks, or the scale of the models, change the conclusion? The paper does not examine these factors.
Questions for authors

Consider the following missing citations which investigate the generalization of ICRL empirically:

"Towards Large-Scale In-Context Reinforcement Learning by Meta-Training in Randomized Worlds", NeurIPS 2025 "Towards general-purpose in-context learning agents." NeurIPS 2023 Foundation Models for Decision Making Workshop

Comments on the paper

1 comment

Thank you for your response!

After giving the paper another read, it is still quite hard to parse and not very clearly written. Regarding scope, I can understand this is a measurement paper, but the measurement is not particularly groundbreaking if the takeaway is: “we recommend also reporting the parameter region covered in training and each evaluation task’s position relative to it." Generally, my concerns don't feel resolved.

For these reasons, I will keep my score. In its current state, the paper needs significant work and should be resubmitted with a more insightful story and conclusive set of results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.