- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies demonstration-free cold starts in frozen in-context reinforcement learning, where an initially empty context and deterministic mean-action deployment can lead to persistent failures as the agent repeatedly conditions on its own episodes. A survey of seven methods across ten locomotion and manipulation families motivates a closer diagnosis of standing and overshooting behaviors. Controlled context swaps and reward relabeling show that a model trapped by its own history can recover with a random trajectory and respond to its task-dependent rewards. The proposed remedy, exploratory prompting (EP), collects one episode with random actions without retraining. AD-lineage models use it as the first episode in their context buffer, while DPT-lineage models compare the returns induced by self-generated and random prompts and retain the random prompt when it performs at least as well. Across four model variants, EP improves mean final return in most family–method pairs, with tradeoffs in cumulative return. The paper compares alternative context and exploration rules, evaluates prompt-dropout fine-tuning as a separate remedy, and provides conditional theoretical analyses of failure and prompt selection.
Critical strengths
-
The controlled diagnosis is the strongest contribution. Context swaps on frozen checkpoints show that changing the history can recover behavior the model can already produce. Relabeling rewards in a random trajectory changes subsequent behavior, providing direct evidence that the learner uses reward information. The deterministic cold-start trigger adds a specific failure case to the earlier observations of misleading self-generated context, which the paper already acknowledges.
-
The experiment coverage is substantial: seven methods are surveyed across ten locomotion and manipulation families, and EP is evaluated on four variants with six training seeds and paired comparisons. Positive mean last-5 gains in 34 of 40 pairs, with hierarchical 95% intervals entirely above zero in 17, support a recurring benefit. The rolling, fixed-prompt, selection-only, and action-noise ablations help assess which parts of the intervention matter.
-
Prompt-dropout fine-tuning is compared with step-matched continued training and evaluated with the original deployment rule. Its improvement after a weak first episode provides useful evidence about continuation from self-generated history. The paper also reports cumulative-return costs, negative results, and cases where simpler context rules perform well.
Critical weaknesses
-
The main results use one fixed EP rule per lineage across task families, but these rules were designed and selected using the same held-out tasks on which their performance is reported. This leaves the gain estimates and ranking against alternatives open to benchmark-level selection bias. New reset draws and the training-task check provide useful robustness evidence, but neither evaluates the selected rules on tasks excluded from both pretraining and rule development. The controlled context interventions remain informative; the material gap concerns independent validation of the chosen rules' advantage on new tasks.
-
EP's algorithmic novelty is relatively limited: it adapts random initial context collection and induced-return prompt selection, both with precedents [1, 2], to demonstration-free cold starts. The theory clarifies possible failures and provides a standard two-candidate explore-then-commit analysis, while support for the particular exploration schedule and retention choices comes chiefly from empirical comparisons. The controlled diagnosis is valuable in its own right. I regard EP's algorithmic novelty as a secondary consideration; the more consequential concern is the lack of independent validation of the selected rules.
-
The stated release excludes the shared evaluation harness, offline data, and checkpoints. Although the protocol descriptions are detailed, this limits independent reproduction of the central comparisons. I regard this as a secondary verification limitation.
[1] Gao et al. Context Shift Reduction for Offline Meta-Reinforcement Learning. NeurIPS, 2023. Paper.
[2] Rietz et al. Prompt Tuning Decision Transformers with Structured and Scalable Bandits. NeurIPS, 2025. Paper.
Questions for authors
-
Can you clarify which EP design decisions used the reported held-out-task results and whether any evaluation tasks were reserved until after the rules were fixed?
-
With the model and other evaluation settings held fixed, how does limiting the AD-lineage deployment buffer to three completed episodes, matching the maximum history count seen during training, affect later stuck rates and EP gains?
-
Does changing only the rewards in a self-generated stuck episode shift subsequent behavior toward the relabeled task, as it does with a random context? This would clarify whether the observed reward sensitivity also holds for the histories that cause the trap.
-
Among trials that retain the random prompt, how often and by how much would the continuing self-chain eventually outperform it? Can the existing paired records characterize this tradeoff beyond the Walker-Mass example?
I’ve raised my rating to 3 (borderline accept). The new evaluation addresses my main concern: with the rules fixed in advance, EP retains most of its final-return gains on tasks used neither for pretraining nor for rule development. I also found the new reward-relabeling experiment informative: changing only the rewards in the agent’s own episode shifted its later behavior toward the new target.