acceptodds
Under review as a conference paper at ICLR 2027

Exploratory Prompting for In-Context Reinforcement Learning

Abstract

In-context reinforcement learning pretrains a sequence model on many tasks' interaction data and deploys it frozen, expecting it to improve on a new task from the episodes it adds to its context. We show that a cold start can trap the agent in that context. Acting with its mean action, the model plays a default episode from an empty context. The context it then writes can repeat or amplify that episode, although the episode's rewards still carry the task. We evaluate seven in-context RL methods on ten task families and dissect the trap in the family where a stuck trial is unambiguous. Context swaps in frozen models show that the model reads the task from the rewards of an episode it did not write. A random episode releases almost every stuck trial, and relabeling its rewards steers the agent toward the relabeled task. Our remedy, exploratory prompting, needs neither retraining nor a demonstration: the agent spends one episode acting at random. When its context is a single prompt episode, it keeps that episode as the prompt only if the return it induces is at least the return its own first episode induces. For four methods, exploratory prompting removes most stuck trials and raises the mean final return in 34 of 40 (family, method) pairs by mean gain, at some cost in cumulative return. Separately, for single-prompt methods, dropping the prompt at random during a short fine-tuning stage also removes most stuck trials, whereas step-matched continued training does not. We argue that part of the reported difference between in-context RL methods reflects how they start, not how well they learn.

Primary area:
reinforcement learning
Keywords:
in-context reinforcement learning, algorithm distillation, decision transformer, offline meta-reinforcement learning, exploration, distribution shift

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_NuAnOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies demonstration-free cold starts in frozen in-context reinforcement learning, where an initially empty context and deterministic mean-action deployment can lead to persistent failures as the agent repeatedly conditions on its own episodes. A survey of seven methods across ten locomotion and manipulation families motivates a closer diagnosis of standing and overshooting behaviors. Controlled context swaps and reward relabeling show that a model trapped by its own history can recover with a random trajectory and respond to its task-dependent rewards. The proposed remedy, exploratory prompting (EP), collects one episode with random actions without retraining. AD-lineage models use it as the first episode in their context buffer, while DPT-lineage models compare the returns induced by self-generated and random prompts and retain the random prompt when it performs at least as well. Across four model variants, EP improves mean final return in most family–method pairs, with tradeoffs in cumulative return. The paper compares alternative context and exploration rules, evaluates prompt-dropout fine-tuning as a separate remedy, and provides conditional theoretical analyses of failure and prompt selection.

Critical strengths
  1. The controlled diagnosis is the strongest contribution. Context swaps on frozen checkpoints show that changing the history can recover behavior the model can already produce. Relabeling rewards in a random trajectory changes subsequent behavior, providing direct evidence that the learner uses reward information. The deterministic cold-start trigger adds a specific failure case to the earlier observations of misleading self-generated context, which the paper already acknowledges.

  2. The experiment coverage is substantial: seven methods are surveyed across ten locomotion and manipulation families, and EP is evaluated on four variants with six training seeds and paired comparisons. Positive mean last-5 gains in 34 of 40 pairs, with hierarchical 95% intervals entirely above zero in 17, support a recurring benefit. The rolling, fixed-prompt, selection-only, and action-noise ablations help assess which parts of the intervention matter.

  3. Prompt-dropout fine-tuning is compared with step-matched continued training and evaluated with the original deployment rule. Its improvement after a weak first episode provides useful evidence about continuation from self-generated history. The paper also reports cumulative-return costs, negative results, and cases where simpler context rules perform well.

Critical weaknesses
  1. The main results use one fixed EP rule per lineage across task families, but these rules were designed and selected using the same held-out tasks on which their performance is reported. This leaves the gain estimates and ranking against alternatives open to benchmark-level selection bias. New reset draws and the training-task check provide useful robustness evidence, but neither evaluates the selected rules on tasks excluded from both pretraining and rule development. The controlled context interventions remain informative; the material gap concerns independent validation of the chosen rules' advantage on new tasks.

  2. EP's algorithmic novelty is relatively limited: it adapts random initial context collection and induced-return prompt selection, both with precedents [1, 2], to demonstration-free cold starts. The theory clarifies possible failures and provides a standard two-candidate explore-then-commit analysis, while support for the particular exploration schedule and retention choices comes chiefly from empirical comparisons. The controlled diagnosis is valuable in its own right. I regard EP's algorithmic novelty as a secondary consideration; the more consequential concern is the lack of independent validation of the selected rules.

  3. The stated release excludes the shared evaluation harness, offline data, and checkpoints. Although the protocol descriptions are detailed, this limits independent reproduction of the central comparisons. I regard this as a secondary verification limitation.

[1] Gao et al. Context Shift Reduction for Offline Meta-Reinforcement Learning. NeurIPS, 2023. Paper.

[2] Rietz et al. Prompt Tuning Decision Transformers with Structured and Scalable Bandits. NeurIPS, 2025. Paper.

Questions for authors
  1. Can you clarify which EP design decisions used the reported held-out-task results and whether any evaluation tasks were reserved until after the rules were fixed?

  2. With the model and other evaluation settings held fixed, how does limiting the AD-lineage deployment buffer to three completed episodes, matching the maximum history count seen during training, affect later stuck rates and EP gains?

  3. Does changing only the rewards in a self-generated stuck episode shift subsequent behavior toward the relabeled task, as it does with a random context? This would clarify whether the observed reward sensitivity also holds for the histories that cause the trap.

  4. Among trials that retain the random prompt, how often and by how much would the continuing self-chain eventually outperform it? Can the existing paired records characterize this tradeoff beyond the Walker-Mass example?

Reviewer_vbkJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies cold-start failures in continuous-control in-context reinforcement learning. Through frozen-model context swaps, reward relabeling, and trajectory analysis, it shows that some models fail to use their self-generated contexts despite being able to infer the task from externally generated episodes. Exploratory prompting spends one episode acting randomly; the DPT variant retains this episode as a prompt when its tested induced return is at least as high as that of the self-generated prompt, while the AD variant inserts it into its buffer. Across four methods and ten task families, the authors report positive mean late-return gains in 34 of 40 pairs, with cumulative-return costs in the DPT lineage.

Critical strengths

S1. A focused and informative diagnosis. Context swaps and reward relabeling hold model weights fixed and directly test whether the model can use task information in the supplied episode. The distinction between standing and overshooting also shows why low return alone is insufficient to characterize the failure. This is a useful empirical contribution beyond proposing random exploration.

S2. Careful reporting of controls, uncertainty, and costs. The paper includes six training seeds, paired comparisons, hierarchical intervals, selection-only and action-noise controls, and step-matched continued-training comparisons for prompt dropout. Exploration and testing are included within the common ten-episode budget. The authors also distinguish positive mean gains from statistically resolved effects and explicitly report cumulative-return losses.

Critical weaknesses

W1. The intervention's generalization is not independently established. The authors state that the context rules were designed and selected using the held-out tasks reported in the main evaluation. New resets on these tasks and checks on training tasks do not provide an untouched task-level test. Moreover, EP is evaluated on four closely related AD/DPT implementations, while the detailed mechanism analysis is concentrated on HalfCheetah-Vel. These choices limit how broadly the reported improvement can be interpreted.

W2. The main experimental results cannot be independently audited from the supplied supplement. The supplement implements the context rules, prompt dropout, and a toy example, but omits the actual evaluation harness, baseline adapters, task maps, checkpoints, and episode records. These artifacts matter because the results depend on correct context construction and paired evaluation across baseline implementations. The toy implementation illustrates the method but does not validate the reported failure rates or gains.

Questions for authors

Q1. Can the selected EP rules be evaluated, without further tuning, on a task split that was not used for rule design? How do the paired late-return and cumulative-return results change?

Q2. DPT EP reduces cumulative return in 18 of 20 pairs, and selection-only achieves higher AUC on nine of ten families. Under what task conditions or deployment horizons should EP be preferred? Please distinguish improvement in late return from improvement in overall interaction efficiency.

Q3. Can the authors provide the evaluation harness, baseline adapters, task maps, configurations, and per-episode records underlying the main tables, together with representative checkpoints or a feasible reconstruction procedure?

Reviewer_eBnHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper asks why some frozen in-context reinforcement learners remain stuck even after receiving informative experience. It tests this through context swaps and reward relabeling, then tries a simple intervention: give the learner a random episode as context. Exploratory prompting is evaluated on four methods across ten task families, with separate experiments on prompt-dropout fine-tuning.

Critical strengths
  1. The context-swap experiments are the strongest part of the paper. Random context releases 59 of 68 stuck trials on the selected checkpoints, and changing the context rewards changes which target the model moves toward. This shows that the model can use task information from another episode even when its own experience fails to change its behavior.

  2. The authors test six training seeds and include action-noise and selection-only controls. They also show where EP hurts performance, which helps explain when someone would actually want to use it.

Critical weaknesses
  1. The context rules were chosen after looking at the reported held-out tasks, so I would be cautious about expecting the same gains on new tasks. The training-task results help because those tasks were not used to choose EP, although the model had seen them during training.

  2. EP also has a cost during the trial. It lowers cumulative return in 18 of 20 DPT-lineage pairs, and selection-only is competitive on that measure. If total return matters more than the final episodes, selection-only may be more attractive. The paper reports this tradeoff, but it should be kept in mind when recommending EP.

Questions for authors

How many context rules did you compare before choosing the final ones, and which choices were made using held-out performance?

Comments on the paper

1 comment

I’ve raised my rating to 3 (borderline accept). The new evaluation addresses my main concern: with the rules fixed in advance, EP retains most of its final-return gains on tasks used neither for pretraining nor for rule development. I also found the new reward-relabeling experiment informative: changing only the rewards in the agent’s own episode shifted its later behavior toward the new target.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.