- Rating
- 4(+2)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
The paper proposes VLM-RB, a simple and practical experience-replay method that uses a frozen VLM to assign semantic scores to replay-buffer sub-trajectories and combines these scores with TD-error-based prioritization. The method is evaluated with four off-policy RL algorithms—DQN, IQN, SAC, and TD3—covering both discrete and continuous control. The empirical results are strong: VLM-RB improves sample efficiency in 23 of 24 reported comparisons, and Section 4.2 provides a particularly useful diagnostic showing that the gains are tied to the pretrained VLM's visual-semantic knowledge rather than simply to another replay weighting mechanism.
I find the motivation clear, the method practical, and the empirical results encouraging. My main concerns are about the contribution. The methodological novelty appears somewhat limited, since the core approach adds a frozen VLM-derived semantic signal to conventional TD-error-based prioritized replay. In addition, comparisons with stronger replay-prioritization methods are limited to DQN, making it difficult to determine whether the reported advantage over these methods extends across the broader set of off-policy algorithms.
Critical strengths
-
The motivation is clear and the method is simple and practical. Using pretrained semantic knowledge to complement TD-error-based replay prioritization is intuitive and potentially useful in visually grounded sparse-reward tasks.
-
Section 4.2 provides a strong diagnostic experiment. By modifying only the visual inputs provided to the VLM while keeping the underlying MDP and the RL agent's observations unchanged, the authors isolate the role of pretrained semantic knowledge. Performance drops under misleading and abstract visualizations, providing good evidence that the improvement genuinely comes from useful VLM priors.
-
The method is evaluated across several RL algorithms. Experiments include DQN, IQN, SAC, and TD3 and cover both discrete and continuous action spaces.
-
The sample-efficiency results are strong and consistent. VLM-RB improves sample efficiency in 23 of 24 reported comparisons and shows particularly large gains on the more difficult long-horizon tasks.
Critical weaknesses
-
The methodological novelty is relatively limited. The core algorithmic change is to obtain a semantic score from a frozen VLM and combine it with TD-error-based prioritized replay. The use of pretrained semantic information is interesting and empirically effective, but the underlying modification to experience replay is fairly simple.
-
The comparison with stronger replay-prioritization methods is incomplete. AER, ERO, and ReLo are evaluated only under DQN, while the broader algorithmic evaluation compares mainly against UER and PER. This makes it difficult to determine whether VLM-RB's advantage over more sophisticated replay approaches extends across different off-policy algorithms.
-
The scope of applicability is inherently limited by the VLM prior. The method performs well when task progress corresponds to visual concepts recognizable by the pretrained VLM, but the paper's own controlled experiments show substantial degradation when those semantics are misleading or removed. This limits applicability to abstract, non-visual, visually unusual, or sufficiently OOD environments. The authors acknowledge this limitation, but it remains an important constraint on the generality of the method.
Questions for authors
-
Claim about replacing heuristic inductive biases. The paper states that VLM-RB replaces heuristic inductive biases with external semantic understanding. However, Equation 2 multiplies the VLM score by the TD error, so the method appears to augment standard statistical prioritization rather than replace it. Could the authors clarify or tone down this claim?
-
Novelty claim around replay buffers. The statement that replay buffers remain unexplored in LLM/VLM-enhanced RL seems too broad. Prior work has already considered language- or foundation-model-related experience reuse mechanisms. The novelty claim may be more precise if framed as using a frozen VLM as an external semantic evaluator for experience prioritization in an off-policy replay buffer.
-
Generality across off-policy algorithms. The paper describes VLM-RB as applicable to “any off-policy algorithm” using a replay buffer. Experiments across four algorithms provide useful evidence of generality, but they do not establish universal compatibility. Could the authors either provide a stronger argument for this claim or narrow it to a broad class of replay-buffer-based off-policy methods?
-
Multiplicative priority function. What motivates multiplying the VLM score and TD error in Equation 2? Since these signals measure different properties of an experience, it would be useful to compare the multiplicative formulation against alternatives such as a weighted additive combination.
-
Sensitivity to VLM false negatives. The VLM score for a clip is computed once and remains fixed, while TD errors continue to evolve throughout training. If an important clip receives a very low VLM score, its prioritized probability may remain very small even when its TD error later becomes large. The uniform component reduces this risk, but could the authors more directly analyze sensitivity to VLM false negatives or low recall?
-
Limited comparison with stronger replay methods. Why are AER, ERO, and ReLo evaluated only with DQN? Since one of the main empirical claims is that VLM-RB improves on existing prioritization strategies, extending these comparisons to SAC, TD3, or IQN would provide stronger evidence that the advantage is not specific to the DQN setting.
-
Scope of applicability. Section 4.2 shows that performance drops substantially when the visual semantics are misleading or abstract, and the paper correctly acknowledges that the approach depends on task progress being visually recognizable. Could the authors discuss more explicitly what classes of environments they expect VLM-RB to work well on, particularly for OOD or visually unusual tasks?
-
Computational overhead. The reported approximately 12% throughput overhead is measured with the RL learner and VLM running on separate GPUs. Could the authors report or discuss the overhead in a single-GPU or resource-constrained setting, where VLM inference directly competes with RL training for compute and memory?
5 comments
Thank you for the rebuttal and additional experiments.
I have a follow up question regarding the false negative (FN): you mention that the VLM misses only 1% of success clips (Table 12). Nevertheless, this table reports a recall of 0.62 for milestone acquisition (0.38 of FN rate). Could you clarify how this relatively low recall affects the robustness argument? Particularly, the random label-corruption is informative, but it may not capture systematic FNs affecting specific task-critical milestones.
Additionally, I believe it would be helpful to report uncertainty estimates (SEM) for the new SAC results.
Overall, I appreciate your efforts addressing my concerns. I have increased my score to 3. I am open to increasing my score further.
Thank you for the follow-up questions. We ran a new experiment to test systematic false negatives on a task-critical milestone.
Regarding false negatives on milestones: Our main results already include these misses, since Table 12 tests the same VLM labeling that Table 1 uses.
- Systematic misses gradually lower VLM-RB performance, down to PER's level. In the new experiment, the VLM first labels each clip. Then, with a fixed probability, we change each positive label on a clip that shows a key pickup but not the goal to negative:
| Key-pickup labels set to negative | Final success | Seeds reaching 0.8 |
|---|---|---|
| None (VLM-RB) | 1.00 ± 0.00 | 5/5 |
| With probability 0.1 | 0.92 ± 0.08 | 4/5 |
| With probability 0.25 | 0.76 ± 0.19 | 4/5 |
| PER | 0.76 ± 0.19 | 4/5 |
- Catching every milestone is not enough. We kept VLM-RB's sampler and replaced the VLM label with the simulator's true labels. The labels for success, key and door miss no milestone, yet the VLM label does better:
| Clip label | Final success | Seeds reaching 0.8 |
|---|---|---|
| Ground truth: success only | 0.40 ± 0.24 | 2/5 |
| Ground truth: success, key and door | 0.60 ± 0.24 | 3/5 |
| VLM, zero-shot | 1.00 ± 0.00 | 5/5 |
Regarding SEM for the SAC results: We have added the SEM to the SAC table in our earlier response. The revised appendix will include it too.
We hope these answers resolve your remaining concerns.
Thank you for the detailed review. As you asked, we also ran AER, ERO, and ReLo with SAC, and VLM-RB outperforms all three.
Regarding novelty (1st weakness): The contribution is the signal that drives prioritization. It flags not only success but also milestones that get no reward. Other replay methods prioritize by TD error, reward, or state similarity, all of which miss such milestones. The §4.2 diagnostic you highlighted shows that the gain depends on the VLM's pretrained semantics. Using the signal is also not straightforward. Using it to prioritize from the first update fails, with success 0. With our mixing schedule, which raises the prioritized share from 0 to 50%, success reaches 1.00 (Table 7, App. F.3). We view the simplicity of our method as an advantage: VLM-RB changes neither the learner's loss nor its neural networks, so it works with existing learners as they are. We show this with DQN, IQN, SAC, and TD3.
Regarding AER, ERO, and ReLo beyond DQN (2nd weakness, 6th question): We ran all three with SAC on Scene-5, the hardest Scene task (mean ± SEM over 5 seeds):
Regarding the wording of three claims (questions 1–3): We changed the text as follows.
Regarding the product form (4th question): The VLM label is binary, so the product works as a gate. Within flagged clips, transitions are ranked by their TD error, as in PER. A weighted sum would add a parameter to tune, and flagged transitions would keep a fixed priority even where the value function is already accurate. App. F.1 tests each factor independently, and each one falls short of the product (Table 4).
Regarding false negatives (5th question): They are rare, and VLM-RB can handle many more than we see in practice (App. F.7–F.8):
Regarding scope (3rd weakness, 7th question): Like any VLM-enhanced RL method, VLM-RB works well when task progress is visually observable. It does not apply where the state has no visual representation, as the paper states (L452–463). It also helps less when the rendering distorts or removes the semantic cues the VLM relies on, such as keys and doors. §4.2 measures this: with misleading or abstract visuals, VLM-RB reduces to PER. As this is a limitation of any VLM-based approach, VLM rewards show the same effect. A CLIP reward on MountainCar works only once the scene has realistic textures (Rocamonde et al., ICLR 2024).
Regarding single-GPU overhead (8th question): The VLM we use in this work is light. The 1B model peaks at 3.8 GiB and scores a 32-frame clip in 0.46 s on an RTX 4090 (Table 9). We run it on a separate GPU, so the measured overhead reflects the algorithm and not the hardware (L458–460). Others do the same: ONI (Zheng et al., RLC 2025) serves its language model from a separate machine, and VLM-RM (Rocamonde et al., ICLR 2024) spreads CLIP inference over four GPUs. A run also does not need the VLM GPU to itself. A Scene run uses at most 42% of one VLM's capacity (0.9 of 2.16 clips/s on the 4090). In a new load test, we fed one VLM the combined peak clip rate of several Scene runs:
So one VLM GPU serves two SAC or three TD3 Scene runs, and each run needs only a half or a third of the extra GPU. We will add this discussion to the appendix, with a note that the runtime overhead depends on the GPU and the setup.
We believe these answers address your concerns, and we welcome follow-up questions and discussion.