acceptodds
Under review as a conference paper at ICLR 2027

VLM-Guided Experience Replay

Abstract

Recent advances in Large Language Models (LLMs) and Vision-Language Models (VLMs) have enabled powerful semantic and multimodal reasoning capabilities, creating new opportunities to enhance sample efficiency, high-level planning, and interpretability in reinforcement learning (RL). While prior work has integrated LLMs and VLMs into various components of RL, the replay buffer, a core component for storing and reusing experiences, remains unexplored. We propose addressing this gap by leveraging VLMs to guide the prioritization of experiences in the replay buffer. Our key idea is to use a frozen, pre-trained VLM as an automated evaluator to identify and prioritize promising sub-trajectories from the agent's experiences. Across scenarios, including game-playing and robotics, spanning both discrete and continuous domains, agents trained with our proposed prioritization method achieve **15-57%** higher average success rates and improve sample efficiency by **35-55%** compared to previous approaches.

Primary area:
reinforcement learning
Keywords:
Reinforcement learning, vision-language models, prioritized experience replay, off-policy learning, sample efficiency

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.25 over 4 reviews, above 96% of the others.
Reviewer_8ga8OpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper proposes VLM-RB, a simple and practical experience-replay method that uses a frozen VLM to assign semantic scores to replay-buffer sub-trajectories and combines these scores with TD-error-based prioritization. The method is evaluated with four off-policy RL algorithms—DQN, IQN, SAC, and TD3—covering both discrete and continuous control. The empirical results are strong: VLM-RB improves sample efficiency in 23 of 24 reported comparisons, and Section 4.2 provides a particularly useful diagnostic showing that the gains are tied to the pretrained VLM's visual-semantic knowledge rather than simply to another replay weighting mechanism.

I find the motivation clear, the method practical, and the empirical results encouraging. My main concerns are about the contribution. The methodological novelty appears somewhat limited, since the core approach adds a frozen VLM-derived semantic signal to conventional TD-error-based prioritized replay. In addition, comparisons with stronger replay-prioritization methods are limited to DQN, making it difficult to determine whether the reported advantage over these methods extends across the broader set of off-policy algorithms.

Critical strengths
  1. The motivation is clear and the method is simple and practical. Using pretrained semantic knowledge to complement TD-error-based replay prioritization is intuitive and potentially useful in visually grounded sparse-reward tasks.

  2. Section 4.2 provides a strong diagnostic experiment. By modifying only the visual inputs provided to the VLM while keeping the underlying MDP and the RL agent's observations unchanged, the authors isolate the role of pretrained semantic knowledge. Performance drops under misleading and abstract visualizations, providing good evidence that the improvement genuinely comes from useful VLM priors.

  3. The method is evaluated across several RL algorithms. Experiments include DQN, IQN, SAC, and TD3 and cover both discrete and continuous action spaces.

  4. The sample-efficiency results are strong and consistent. VLM-RB improves sample efficiency in 23 of 24 reported comparisons and shows particularly large gains on the more difficult long-horizon tasks.

Critical weaknesses
  1. The methodological novelty is relatively limited. The core algorithmic change is to obtain a semantic score from a frozen VLM and combine it with TD-error-based prioritized replay. The use of pretrained semantic information is interesting and empirically effective, but the underlying modification to experience replay is fairly simple.

  2. The comparison with stronger replay-prioritization methods is incomplete. AER, ERO, and ReLo are evaluated only under DQN, while the broader algorithmic evaluation compares mainly against UER and PER. This makes it difficult to determine whether VLM-RB's advantage over more sophisticated replay approaches extends across different off-policy algorithms.

  3. The scope of applicability is inherently limited by the VLM prior. The method performs well when task progress corresponds to visual concepts recognizable by the pretrained VLM, but the paper's own controlled experiments show substantial degradation when those semantics are misleading or removed. This limits applicability to abstract, non-visual, visually unusual, or sufficiently OOD environments. The authors acknowledge this limitation, but it remains an important constraint on the generality of the method.

Questions for authors
  1. Claim about replacing heuristic inductive biases. The paper states that VLM-RB replaces heuristic inductive biases with external semantic understanding. However, Equation 2 multiplies the VLM score by the TD error, so the method appears to augment standard statistical prioritization rather than replace it. Could the authors clarify or tone down this claim?

  2. Novelty claim around replay buffers. The statement that replay buffers remain unexplored in LLM/VLM-enhanced RL seems too broad. Prior work has already considered language- or foundation-model-related experience reuse mechanisms. The novelty claim may be more precise if framed as using a frozen VLM as an external semantic evaluator for experience prioritization in an off-policy replay buffer.

  3. Generality across off-policy algorithms. The paper describes VLM-RB as applicable to “any off-policy algorithm” using a replay buffer. Experiments across four algorithms provide useful evidence of generality, but they do not establish universal compatibility. Could the authors either provide a stronger argument for this claim or narrow it to a broad class of replay-buffer-based off-policy methods?

  4. Multiplicative priority function. What motivates multiplying the VLM score and TD error in Equation 2? Since these signals measure different properties of an experience, it would be useful to compare the multiplicative formulation against alternatives such as a weighted additive combination.

  5. Sensitivity to VLM false negatives. The VLM score for a clip is computed once and remains fixed, while TD errors continue to evolve throughout training. If an important clip receives a very low VLM score, its prioritized probability may remain very small even when its TD error later becomes large. The uniform component reduces this risk, but could the authors more directly analyze sensitivity to VLM false negatives or low recall?

  6. Limited comparison with stronger replay methods. Why are AER, ERO, and ReLo evaluated only with DQN? Since one of the main empirical claims is that VLM-RB improves on existing prioritization strategies, extending these comparisons to SAC, TD3, or IQN would provide stronger evidence that the advantage is not specific to the DQN setting.

  7. Scope of applicability. Section 4.2 shows that performance drops substantially when the visual semantics are misleading or abstract, and the paper correctly acknowledges that the approach depends on task progress being visually recognizable. Could the authors discuss more explicitly what classes of environments they expect VLM-RB to work well on, particularly for OOD or visually unusual tasks?

  8. Computational overhead. The reported approximately 12% throughput overhead is measured with the RL learner and VLM running on separate GPUs. Could the authors report or discuss the overhead in a single-GPU or resource-constrained setting, where VLM inference directly competes with RL training for compute and memory?

5 comments

Thank you for the detailed review. As you asked, we also ran AER, ERO, and ReLo with SAC, and VLM-RB outperforms all three.

Regarding novelty (1st weakness): The contribution is the signal that drives prioritization. It flags not only success but also milestones that get no reward. Other replay methods prioritize by TD error, reward, or state similarity, all of which miss such milestones. The §4.2 diagnostic you highlighted shows that the gain depends on the VLM's pretrained semantics. Using the signal is also not straightforward. Using it to prioritize from the first update fails, with success 0. With our mixing schedule, which raises the prioritized share from 0 to 50%, success reaches 1.00 (Table 7, App. F.3). We view the simplicity of our method as an advantage: VLM-RB changes neither the learner's loss nor its neural networks, so it works with existing learners as they are. We show this with DQN, IQN, SAC, and TD3.

Regarding AER, ERO, and ReLo beyond DQN (2nd weakness, 6th question): We ran all three with SAC on Scene-5, the hardest Scene task (mean ± SEM over 5 seeds):

MethodBest successSteps to PER's best
PER0.62 ± 0.13n/a
AER0.00 ± 0.00never
ERO0.88 ± 0.05574K ± 54K
ReLo0.86 ± 0.05828K ± 67K
VLM-RB0.94 ± 0.02494K ± 36K
As we expected, VLM-RB leads on every measure and is the first to reach PER's best.

Regarding the wording of three claims (questions 1–3): We changed the text as follows.

  • Replacing heuristics (L88, L99–102, L446): "complement statistical prioritization with external semantic understanding."
  • Replay buffers (abstract, L14): The abstract now says the replay buffer "remains underexplored". §5 already uses this word (L438). The next sentence already gives the framing you suggest: "a frozen, pre-trained VLM as an automated evaluator to identify and prioritize promising sub-trajectories" (L16–17).
  • Generality (L104, L166): VLM-RB only changes which transitions are sampled, and it needs nothing from the learner beyond what PER needs. The text now reads "off-policy algorithms that sample from a replay buffer". We show it with value-based, distributional, and actor-critic learners.

Regarding the product form (4th question): The VLM label is binary, so the product works as a gate. Within flagged clips, transitions are ranked by their TD error, as in PER. A weighted sum would add a parameter to tune, and flagged transitions would keep a fixed priority even where the value function is already accurate. App. F.1 tests each factor independently, and each one falls short of the product (Table 4).

Regarding false negatives (5th question): They are rare, and VLM-RB can handle many more than we see in practice (App. F.7–F.8):

  • The VLM misses only 1% of success clips (Table 12).
  • Swapping up to 25% of the positive labels with negative ones keeps success at 0.96–1.00 (Table 13).
  • By design, a clip the VLM wrongly labels negative is still replayed. It keeps at least half its uniform replay rate (Eq. 5).

Regarding scope (3rd weakness, 7th question): Like any VLM-enhanced RL method, VLM-RB works well when task progress is visually observable. It does not apply where the state has no visual representation, as the paper states (L452–463). It also helps less when the rendering distorts or removes the semantic cues the VLM relies on, such as keys and doors. §4.2 measures this: with misleading or abstract visuals, VLM-RB reduces to PER. As this is a limitation of any VLM-based approach, VLM rewards show the same effect. A CLIP reward on MountainCar works only once the scene has realistic textures (Rocamonde et al., ICLR 2024).

Regarding single-GPU overhead (8th question): The VLM we use in this work is light. The 1B model peaks at 3.8 GiB and scores a 32-frame clip in 0.46 s on an RTX 4090 (Table 9). We run it on a separate GPU, so the measured overhead reflects the algorithm and not the hardware (L458–460). Others do the same: ONI (Zheng et al., RLC 2025) serves its language model from a separate machine, and VLM-RM (Rocamonde et al., ICLR 2024) spreads CLIP inference over four GPUs. A run also does not need the VLM GPU to itself. A Scene run uses at most 42% of one VLM's capacity (0.9 of 2.16 clips/s on the 4090). In a new load test, we fed one VLM the combined peak clip rate of several Scene runs:

Runs sharing one VLM (at peak demand)VLM busyMedian score delay
1 SAC Scene-5 run42%0.47 s
2 SAC or 3 TD3 Scene-5 runs82%1.5 s

So one VLM GPU serves two SAC or three TD3 Scene runs, and each run needs only a half or a third of the extra GPU. We will add this discussion to the appendix, with a note that the runtime overhead depends on the GPU and the setup.

We believe these answers address your concerns, and we welcome follow-up questions and discussion.

Thank you for the rebuttal and additional experiments.

I have a follow up question regarding the false negative (FN): you mention that the VLM misses only 1% of success clips (Table 12). Nevertheless, this table reports a recall of 0.62 for milestone acquisition (0.38 of FN rate). Could you clarify how this relatively low recall affects the robustness argument? Particularly, the random label-corruption is informative, but it may not capture systematic FNs affecting specific task-critical milestones.

Additionally, I believe it would be helpful to report uncertainty estimates (SEM) for the new SAC results.

Overall, I appreciate your efforts addressing my concerns. I have increased my score to 3. I am open to increasing my score further.

Thank you for the follow-up questions. We ran a new experiment to test systematic false negatives on a task-critical milestone.

Regarding false negatives on milestones: Our main results already include these misses, since Table 12 tests the same VLM labeling that Table 1 uses.

  • Systematic misses gradually lower VLM-RB performance, down to PER's level. In the new experiment, the VLM first labels each clip. Then, with a fixed probability, we change each positive label on a clip that shows a key pickup but not the goal to negative:
Key-pickup labels set to negativeFinal successSeeds reaching 0.8
None (VLM-RB)1.00 ± 0.005/5
With probability 0.10.92 ± 0.084/5
With probability 0.250.76 ± 0.194/5
PER0.76 ± 0.194/5
  • Catching every milestone is not enough. We kept VLM-RB's sampler and replaced the VLM label with the simulator's true labels. The labels for success, key and door miss no milestone, yet the VLM label does better:
Clip labelFinal successSeeds reaching 0.8
Ground truth: success only0.40 ± 0.242/5
Ground truth: success, key and door0.60 ± 0.243/5
VLM, zero-shot1.00 ± 0.005/5

Regarding SEM for the SAC results: We have added the SEM to the SAC table in our earlier response. The revised appendix will include it too.

We hope these answers resolve your remaining concerns.

Reviewer_DFSsOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes VLM-RB. A frozen video VLM (Perception-LM-1B) scores non-overlapping 32-frame rendered clips with a binary "is goal satisfaction visible" prompt. The score is thresholded, multiplied by |TD error|, and used in a sampler that mixes prioritized and uniform replay, with the prioritized share rising linearly from 0 to 0.5. Scoring runs asynchronously, so the learner never waits on the VLM.

Critical strengths

The method changes no loss and needs no VLM fine-tuning. Asynchronous scoring keeps the throughput cost to about 12%. It improves results across four algorithms with discrete and continuous actions, and the largest gains come on the hardest settings (DoorKey-16x16, Scene-5).

Critical weaknesses
  1. The score works as a success detector. The prompt asks whether goal satisfaction is visible. Table 12 shows the binary label has 0.99 recall on strict success, and most of its false positives are milestone events (key pickup, door opening). In practice the method upweights clips where success or a subgoal happens. And in DoorKey, the reward and the symbolic state changes already reveal it. In Scene, the shaped reward ϕ\phi explicitly counts satisfied subgoals.
  2. The schedule is a major driver on its own. Table 7 shows VLM-RB itself collapses to near-zero success under pure prioritization. Some of the gap with PER may therefore come from the schedule rather than the VLM. The fair control is TD-only priority with the same λ schedule and the same IS setting.
Questions for authors
  1. What happens if p^VLM is replaced by a ground-truth milestone/success indicator from the simulator, or, in Scene, by a priority based on Δϕ\Delta \phi ? If VLM-RB roughly matches the oracle, and a Δϕ\Delta \phi priority is competitive, the contribution reduces to "success detection helps replay."
  2. Do the conclusions in Table 1 hold with final-step ASR, or with IQM plus bootstrap CIs, instead of the per-seed best over training?
2 comments

Thank you for the review. You asked whether the VLM is more than a success detector, and whether the gain comes from the schedule. We ran two controls you proposed: the simulator's true labels in place of the VLM label, and TD-only priority with the same schedule. We also computed final-step statistics for Table 1.

Regarding the success detector and the oracle (1st weakness, 1st question): We ran the test you proposed. We kept VLM-RB's sampler and replaced the VLM label with the simulator's true labels for milestones and successful completions (DoorKey-16×16, DQN, 5 seeds):

Clip labelFinal successSeeds reaching 0.8
Ground truth: success only0.40 ± 0.242/5
Ground truth: success, key and door0.60 ± 0.243/5
VLM, zero-shot1.00 ± 0.005/5

The zero-shot VLM label beats both true labels, so it carries more than a success signal.

  • The VLM flags more than success. Of the flagged clips that do not show the goal, 74% show the key picked up, or the door opened (App. F.7). DoorKey's reward does not mark these events (App. B), and our prompt never mentions them (App. C). On Scene, too, most flagged clips come from attempts that never reach the goal (about 93% of flagged transitions, Table 8).
  • Recognizing the milestones matters. When the key is drawn as lava and the door as a box, the VLM flags fewer key and door events, and VLM-RB performs no better than PER (Misleading variant, Figs. 5 and 10b).

Regarding the schedule (2nd weakness): We ran the control you suggested: TD-only priority with VLM-RB's warm-up schedule and importance-sampling setting (DoorKey-16×16, DQN, 5 seeds).

Priority, same scheduleFinal successSteps to 0.8 successSeeds reaching 0.8
TD error only0.88 ± 0.12465K ± 21K4/5
VLM-RB1.00 ± 0.00354K ± 25K5/5

With the schedule matched, the VLM label gets every seed to 0.8 success, and 24% sooner.

Regarding final-step metrics (2nd question): Yes, the conclusions hold. Across all 12 configurations (5 seeds each), the IQM of final success with 95% bootstrap CIs is 1.00 [0.99, 1.00] for VLM-RB, 0.87 [0.74, 0.95] for PER, and 0.68 [0.54, 0.81] for UER. VLM-RB's interval does not overlap with either baseline.

We believe these answers address your questions, and we welcome any follow-up questions or discussion.

The authors' responses resolve my concerns. I maintain my positive recommendation.

Reviewer_fEKtOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes VLM-RB, a replay-prioritization scheme for off-policy RL in which a frozen video VLM scores rendered sub-trajectories of the agent's experience. Episodes are split into non-overlapping 32-frame clips; Perception-LM-1B is asked a binary question ("Does this clip contain a clear instance of goal satisfaction anywhere in it? ... Answer: Yes or No"), and the Yes/No probability ratio is thresholded to give a binary clip score p_VLM, which is propagated to all transitions in the clip. The priority is p_VLM·|δ| (TD error), and each batch mixes prioritized and uniform sampling with a mixing weight for training.

Experiments on MiniGrid DoorKey (8×8, 12×12, 16×16) and OGBench Scene (tasks 3–5) with DQN, IQN, TD3, and SAC (5 seeds) show that VLM-RB matches or exceeds uniform replay (UER) and PER in all configurations and improves sample efficiency in 23/24 comparisons.

Critical strengths
  1. Using a frozen VLM to prioritize replay is easy to implement, changes no loss function, and can be added to any off-policy algorithm. The asynchronous scoring design and one-time scoring of each clip (no priority staleness) are practical engineering choices.
  2. Improvements are consistent across four algorithms and both discrete and continuous control, and grow with task difficulty.
  3. Extensive ablations are included in the paper, which makes the work quite thorough.
Critical weaknesses
  1. Missing related work: The motivation is explicitly about delayed credit assignment in long-horizon, sparse-reward, goal-reaching tasks (e.g., "delayed credit assignment preventing the reward signal from propagating to these earlier states"). Standard remedies, hindsight experience replay (HER, Andrychowicz et al., 2017) for goal-reaching tasks and reward-redistribution methods such as RUDDER (Arjona-Medina et al., 2019), are neither compared nor discussed. HER is particularly relevant for OGBench, and Hindsight Goal Ranking (Luu & Yoo, 2021), which the paper cites, is a replay-prioritization method for sparse-reward environments.
  2. The binary clip score is copied to all 32 transitions of the clip. A successful clip therefore upweights every transition in it equally, including irrelevant or suboptimal ones, while transitions leading up to a success but falling in the preceding clip receive no boost. This is a coarse, fixed-window form of credit assignment. The multiplication by |δ| partially refines it, but the paper does not analyze which transitions within positive clips are actually replayed more, nor the sensitivity to clip length (L = 32 is fixed and "not tuned"). A sliding-window or per-transition attribution is mentioned as future work, but some evidence on clip-length sensitivity is needed.
  3. Limited novelty : Prioritizing replay by an external success/progress signal is a modest extension of PER and of replay schemes based on returns or success. The main novelty is the source of the signal (a frozen VLM), and the paper's own analysis suggests that this signal is largely a success detector. The contribution would be clearer if the paper demonstrated a case where the VLM provides information that cannot be obtained from rewards, e.g., tasks with no reward at the milestones, or settings where success is visually apparent but the reward is delayed or noisy.

To the authors: My current rating is largely due to novelty concerns. If this is addressed during rebuttal I am willing to improve my rating.

Questions for authors
  1. How does VLM-RB compare to prioritizing reward-bearing clips or high-return episodes (e.g., self-imitation learning, Episodic Backward Update), and to n-step/λ-return variants of the baselines?
  2. How does VLM-RB compare to using the same VLM signal as a reward bonus (e.g., VLM-RM-style shaping), and to HER on OGBench?
5 comments

Thank you for the review and for clarifying what would change your rating. We start there. We added two new experiments: an oracle-label control and a clip-length sweep.

Regarding novelty (3rd weakness): As you note, the novelty is the source of the priority signal. Other methods prioritizing useful experience need a hand-designed signal, such as an object's trajectory energy (Zhao & Tresp, 2018) or a goal-conditioned agent (HGR). VLM-RB gets its signal from a frozen VLM, zero-shot, with one generic prompt per domain. This signal is more than a success detector. In a new oracle control (the last point below), it does better than the simulator's true labels of success and milestones.

Another novel contribution is where the signal enters the learner. Earlier VLM methods mostly used the model in the reward, for exploration or as the policy (§5). VLM-RB only changes which transitions are replayed. A wrong label can change how often a transition is replayed, but not the reward the agent maximizes. Even when we swap up to 25% of the positive labels with negative ones, success stays at 0.96–1.00 (Table 13).

A task with no reward at the milestones. Take DoorKey, for example. The agent gets a reward only at the final goal. Picking up the key and opening the door gives no reward (App. B). A pure success detector would ignore these events. The VLM does not:

  • It flags the milestones. Of the flagged clips that don't show the goal, 74% show the key being picked up or the door being opened (App. F.7).
  • Recognizing the milestones matters. When the key is drawn as lava and the door as a box, the VLM flags fewer key and door events, and VLM-RB performs no better than PER (Misleading variant, Figs. 5 and 10b). So VLM-RB uses information already encoded in the VLM about keys and doors. Our prompt never mentions them (App. C).
  • Detecting these events is not enough. We kept the sampler and replaced the VLM label with the simulator's true labels for milestones and successful task completion (DoorKey-16×16, DQN, 5 seeds):
Clip labelFinal successSeeds reaching 0.8
Ground truth: success only0.40 ± 0.242/5
Ground truth: success, key and door0.60 ± 0.243/5
VLM, zero-shot1.00 ± 0.005/5

Regarding return-based replay and n-step returns (1st question):

  • SIL and EBU learn from the rewards the agent has seen. SIL imitates actions with higher-than-expected returns, and EBU passes rewards back through an episode. If the agent picks up the key but misses the goal, the episode has no reward, so SIL and EBU learn nothing from it. The VLM still flags the key pickup, so VLM-RB replays that clip more often.
  • n-step and λ-returns change the learning target. VLM-RB changes which transitions are replayed, so the two can be combined. All methods in our comparisons use 1-step targets, so the differences come from sampling alone.

Regarding HER, RUDDER and HGR (1st weakness): HER relabels goals, and RUDDER moves a delayed reward back to the steps that caused it. Both change the learning target. VLM-RB only changes which transitions are sampled, so it can be combined with either. Like SIL and EBU, RUDDER needs a reward to work with. If the agent picks up the key but misses the goal, RUDDER has no reward to move back, but the VLM still flags the key pickup. HER and HGR also need a goal-conditioned agent. VLM-RB does not, and our agents are not goal-conditioned: each run learns one fixed task. The HER paper itself notes that HER and prioritized replay can be combined (Andrychowicz et al., 2017, §5), and Chuck et al. (ICLR 2025) use both.

Regarding a VLM reward bonus (2nd question): We use the VLM for replay rather than reward for two reasons:

  • A reward bonus can change the optimal policy. As a bonus, a wrong label changes the return the agent maximizes (Ng et al., 1999). As a priority, it only changes how often a transition is replayed. So even when we swap up to 25% of the positive labels with negative ones, success stays at 0.96–1.00 (Table 13).
  • A reward bonus is computationally expensive and hard to assign. A VLM reward such as VLM-RM (Rocamonde et al., ICLR 2024) calls the model on every frame. VLM-RB calls it once per 32-frame clip. The VLM also judges a whole clip, so it is unclear which of its 32 steps should get the reward. As a priority, the answer simply marks the clip.

Regarding one score for all 32 frames (2nd weakness): We ran the clip-length ablation (DoorKey-16×16, VLM-only priority, 5 seeds):

Clip lengthFinal successSteps to 0.8 successSeeds reaching 0.8
8 frames0.12 ± 0.12690K1/5
16 frames1.00 ± 0.00482K ± 55K5/5
32 frames (ours)1.00 ± 0.00382K ± 24K5/5

An 8-frame clip is too short for the VLM to judge progress (Fig. 3). A 64-frame clip requires 16,384 video tokens, exceeding the VLM's context limit of 11,475.

Within a positive clip, transitions are not upweighted equally. We propagate the clip's VLM score to its transitions and multiply it by each transition's latest TD error (Eq. 2). The VLM score selects the clips, and the TD error ranks the transitions inside them. The TD error marks where the value function is still inaccurate (L215–216), so transitions whose value is already accurate get a low priority.

  • The TD error matters. On Scene-5, the hardest manipulation task, VLM-RB reaches PER's best success in 34% fewer steps than the clip score alone (Table 4).
  • Earlier clips that show progress are flagged too. The VLM flags the clips where the key is picked up, or the door is opened, so these clips are also replayed more often.

We would value your view on whether this addresses your concerns.

As the discussion period comes to an end, do our replies address the concerns behind your current assessment? If so, please take this into account in your final evaluation. If any point remains open, we are happy to address it.

Reviewer_e6agOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose and evaluate the use of vision-language models for experience prioritization in reinforcement learning. In particular, the authors present VLM-RB, a method that uses a VLM to score an agent's previously experienced trajectories and in turn uses these scores to select transitions to use for policy updates in RL.

The authors compare VLM-RB to a range of other experience prioritization methods in two domains, one discrete and one continuous. Compared to these other prioritization methods, VLM-RB shows near uniform improvements in sample efficiency and final (end-of-training) performance. Furthermore, VLM-RB is shown to not significantly increase training time.

Critical strengths

Presentation and elegance

Overall, the paper is well-presented. The authors clearly motivate their work in the context of existing work on RL, introduce the optimal detail in terms of preliminaries, clearly explain their method, and present their experimental results clearly. The paper is highly polished and was easy to understand.

Beyond presentation, the framework is also elegant. The method the authors present is simple and natural. The authors do not over-complicate VLM-RB, and the design choices they make, such as uniform mixing or asynchronous deployment of the VLM, are well-justified.

Convincing results

Beyond presentation, the results of the paper appear rigorous. Indeed, the presented method performs very well on the presented baselines, and the detailed implementation information the authors provide makes these results more convincing.

Together, these qualities of the paper make for a work that the ICLR community would be interested in and find easy to digest.

Critical weaknesses

Comparisons to other VLM-enabled RL methods

The authors currently limit their comparison of VLM-RB to other experience-prioritization methods, and this choice limits the impact of the work.

In particular, one could argue that VLM-enabled replay prioritization methods could be replaced by other VLM-enabled RL components. For example, one of the authors' core motivations for using VLMs for experience replay is that conventional prioritization methods struggle with sparse-reward tasks. However, as the authors detail in their related work, existing VLM-enabled RL methods also allow automatic synthesis of dense reward functions. As such, one could potentially replace VLM-RB with a VLM-based reward densification method. Ultimately, prioritization methods are one many methods to shape the stream of experience the agent uses for learning, and as such, prioritization methods should also be compared against other methods that shape this experience stream. While the authors discuss that existing VLM-enabled RL methods mainly focus on shaping the agent's rewards, policies and exploration, this discussion fails to address the broader role of experience-replay in RL.

Ultimately, the existing validation is reasonable if the authors want to validate the claim that VLMs improve experience-replay in RL. However, with more thorough comparisons, the authors could make more impactful claims, for example, that VLM-based experience replay is a key component for improving performance in VLM-enabled RL more generally.

Limited evaluation

Some experiments in the paper could also be improved.

Firstly, the work is also limited by the small number of domains, and would be made more convincing by extending the use of VLM-RB beyond DoorKey and scene from OGBench.

Also, while the authors demonstrate that VLM-RB only leads to a 12% throughput overhead, this discussion fails to acknowledge the compute requirements of deploying a VLM on a separate GPU. Indeed, the discussion in Appendix F.12 seems to suggest that the authors use a separate GPU for VLM inference. The paper would benefit from an acknowledgement and measurement of this extra compute over-head.

Questions for authors
  1. Do the authors have any insight into how VLM-guided replay compares to other VLM-enabled RL methods? Obviously, such comparisons are difficult to implement in a rebuttal period. However, I would be curious as to whether the authors can provide any insight based on existing literature? Do existing VLM-enabled RL methods evaluate on any common tasks to the present paper? How do performance and compute requirements compare?
  2. Do the authors have any preliminary results on other domains they can present?
  3. Do the authors have any metrics on the raw increase in compute required for deploying the VLM on the separate GPU?
4 comments

Thank you for the review. We start by comparing VLM-RB with other VLM-enabled RL methods.

Regarding other VLM-enabled RL methods (1st weakness, 1st question): Of the places a VLM can be used in the RL training process, the replay buffer is the most forgiving in terms of model errors, and very efficient by only calling a small model once per clip:

Where the model is usedModel calls per environment stepAn error changes
Reward from VLM scores (VLM-RM, VLC)One per stepReward
Reward model from VLM feedback (RL-VLM-F, ERL-VLM)About one per 50 steps (10,000 labels per 1M steps, two calls each), large APIsReward
LLM-written reward code (Eureka, REvolve)Under one: tens to hundreds of reward candidates per run, each needs a full RL runReward
Policy (SayCan, RT-2)One per step, or per short chunk of steps, very large modelsActions
Replay (VLM-RB)One per clip (32 steps in our runs), 1B modelReplay sample frequency only
A VLM reward changes the MDP, so its errors can change the optimal policy (Ng et al., 1999). VLM-RB leaves the MDP unchanged, so it can be added on top of any of these methods.

Regarding raw compute (3rd question): The VLM we use in this work is light. The 1B model peaks at 3.8 GiB and scores a 32-frame clip in 0.46 s on an RTX 4090 (Table 9). If we count its GPU in full, VLM-RB uses about 2.3× PER's GPU time per step (Table 15). But it reaches PER's best in 35–42% fewer steps (Table 17), so the total cost is about 1.3–1.5× PER's GPU-hours. A run also does not need the VLM GPU to itself. A Scene run uses at most 42% of one VLM's capacity (0.9 of 2.16 clips/s on the 4090). In a new load test, we fed one VLM the combined peak clip rate of several Scene runs:

Runs sharing one VLM (at peak demand)VLM busyMedian score delay
1 SAC Scene-5 run42%0.47 s
2 SAC or 3 TD3 Scene-5 runs82%1.5 s

Two Scene runs can share one VLM GPU, and the total cost drops to about 1.1× PER's. We will add this discussion to the appendix, with a note that the runtime overhead depends on the GPU and the setup.

Regarding more domains (2nd question): We would be glad to hear which domains you have in mind. A new domain would require full training runs for every learner and baseline, so we are not sure we can finish them within the discussion period. The main requirement for a new domain is that the VLM can recognize task progress in the rendered frames. App. F.10 checks this on a third task. On LavaCrossing, the frozen VLM flags the gap crossing before the critic learns its value (Fig. 9). Our main results cover four learners and two domains, each at three difficulty levels. That is 12 settings with 5 seeds each, plus controls on visual semantics (Fig. 5) and on the VLM family (App. F.6).

We believe these answers support your view that the work will interest the ICLR community, and we appreciate the sentiment. We welcome any follow-up discussion.

Reviewer_e6agOpenReview ↗

Response to author's reply

I thank the authors for their clarification and plan to maintain my positive recommendation.

My concerns are mostly addressed, though I have left some minor comments below.

Regarding other VLM-enabled RL methods

The detailed description of different VLM-enabled RL methods was helpful for clarifying VLM-RB's benefits. I have two more comments below.

Firstly, I think this table could be clearer in comparing the VLM-call requirements. For example, under methods that build a "Reward model from VLM feedback", the authors count the number of VLM calls as "Thousands, to large APIs". I assume this metric measures the calls/RL run. However, VLM-RB is listed as requiring a VLM-call "once per clip". If a VLM-RB run involves scoring 1000s or possibly 10000s of clips, would the VLM-usage not also be comparable to other methods? In general, I encourage the authors to make it easier to compare these various methods, by making sure all measurements are either per-step or per-run/are in the same units.

I also still believe numerical comparison of the success rates/throughput of VLM-RB against these existing methods would be valuable. However, if there is not an easy way to make this comparison in the rebuttal period, I understand. Again, I still plan to maintain my positive recommendation.

Regarding raw compute

I thank the authors for the extra quantification of raw-GPU-hour metrics.

Regarding new domains

I fully understand that new domains are not feasible in the rebuttal period, if the authors did not originally plan/start these experiments.

In any case, in terms of new domains, I think more dynamic environments would be more valuable. DoorKey tests VLM-RB's performance in environments where the agent needs to complete a sequence of tasks. While OGBench/scene is a continuous control task it still fundamentally involves an agent completing a sequence of tasks and the reward is still success-based. I would be curious as to whether VLM-RB would be as successful in environments where agents need to find sequences of actions that work well in an environment to perform well in a game or to keep an unstable system stable. Domains such as Atari would be valuable for this purpose.

Thank you for the follow-up comments.

Regarding other VLM-enabled RL methods: We have updated the table in our earlier response so that every row gives model calls per environment step. As in our first reply, the main difference is not the number of calls but what a model error changes.

A VLM reward or a VLM policy calls the model about once per step, and VLM-RB once per clip (32 frames or more, depending on the VLM's context size). Methods that learn a reward model from VLM feedback call it about as often as VLM-RB. RL-VLM-F and ERL-VLM ask Gemini for about 10,000 labels in a 1M-step MetaWorld run, at two calls per label, or about one call per 50 steps. But each label trains a reward model that sets the reward at every step, so label errors enter the reward the agent maximizes. VLM-RB leaves the reward and the MDP unchanged: a wrong label only changes how often a transition is replayed. As a result, VLM-RB is robust to label errors. Swapping up to 25% of the positive labels with negative ones, which adds both false negatives and false positives, keeps success at 0.96–1.00 (Table 13).

This makes these methods complementary to VLM-RB. In an off-policy agent trained with a VLM reward, VLM-RB would replace PER, not the VLM reward. For this reason, we compare VLM-RB with the replay methods: PER, AER, ERO and ReLo.

Regarding new domains: Within the discussion period, we could test whether our VLM sees progress in the Atari games you suggest. VLM-RB replays clips with high VLM scores more often, so we checked where the top-scored clips come from. We scored 32-frame clips of public trained PPO agents and of random play, with the same 1B model and prompt as in our MiniGrid runs. We ranked all clips by VLM score. The table shows where the top half comes from (50/50 if the VLM could not tell them apart):

GameTrained agentRandom play
Breakout84%16%
Enduro72%28%
Freeway61%39%
Kangaroo59%41%

In every game, most of the clips with the highest VLM scores come from the trained agent. Like any VLM-enhanced RL method, VLM-RB works well when task progress is visually observable.

We hope these additions address your remaining concerns.

On other VLM-enabled RL methods: I agree with the authors' insight on VLM-RB's robustness compared to reward-based methods.

I would like to remark that an ideal comparison would involve numerical demonstration that label errors in ERL-VLM/RL-VLM-F lead to policy failures/lower success rates compared to VLM-RB. Indeed, ultimately, with limited computational resources, an RL practitioner must choose where to use the VLM and may have to choose between various VLM-enabled RL methods. That said, I also understand that this comparison may be difficult without reimplementing these methods.

Overall, the authors have addressed my concerns to the extent feasible in a tight rebuttal period. I appreciate the clarified comparison and the analysis of VLM-RB's ability to identify task progress in the atari domain as well.

Comments on the paper

1 comment

We thank the reviewers for the discussion.

Reviewers find VLM-RB simple and practical (e6ag, fEKt, 8ga8), agree that we provide "good evidence that the improvement genuinely comes from useful VLM priors" (8ga8), and the paper is "a work that the ICLR community would be interested in and find easy to digest" (e6ag).

Some reviewers asked whether the VLM is more than a success detector. New ablations confirm that it is: the VLM flags progress the reward misses and beats the simulator's own labels. New baselines, robustness tests, compute measurements, and an Atari analysis answer the remaining points.

All four reviewers responded positively to our replies.

The revised PDF, with changes in blue, changes the wording of three claims (8ga8), adds the SEM for the SAC results (8ga8), adds an appendix discussion of the compute cost (e6ag, 8ga8), qualifies the label-error claim (fEKt), and adds related work on credit assignment (fEKt).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.