Prompt-Driven Exploration

Reinforcement learning is the standard way to post-train foundation models, but progress is bottlenecked by exploration. The usual approach perturbs sampled actions, which only moves the policy locally, yet hard tasks often require strategies very different from what the policy currently does. This is especially problematic when fine-tuning starts from a weak policy with low success rates and little useful behavior to refine. Many foundation-model policies, however, are conditioned on a natural-language prompt, and varying the prompt produces qualitatively different behaviors from the same policy. This suggests a different angle: perturb the prompt instead of the action. Our key insight is that a distribution over prompts induces a distribution over policies, with structure already shaped by the pretrained language prior. This turns exploration into a familiar problem: posterior sampling, where we maintain a belief over which prompts elicit success, sample from it to drive rollouts, and update it from what we observe. A vision-language model fits this role naturally, both proposing plausible prompts and reasoning about past rollouts to shift the distribution toward ones more likely to succeed. We call this Prompt-Driven Exploration (PDE). On hard tasks where action-space exploration fails to make progress, PDE bootstraps from nearly zero success to high success rates with far fewer environment interactions.