acceptodds
Under review as a conference paper at ICLR 2027

MARVL: Multi-Stage Guidance for Reinforcement Learning via Vision-Language Models

Abstract

Designing dense reward functions is pivotal for efficient Reinforcement Learning (RL). However, most dense rewards rely on manual engineering, which fundamentally limits the scalability and automation of reinforcement learning. While Vision-Language Models (VLMs) offer a promising path to reward design, naïve VLM rewards often misalign with task progress, struggle with spatial grounding, and show limited understanding of task semantics. To address these issues, we propose **MARVL**—**M**ulti-st**A**ge guidance for **R**einforcement learning via **V**ision-**L**anguage models. MARVL fine-tunes a VLM for spatial and semantic consistency and decomposes tasks into multi-stage subtasks with task direction projection for trajectory sensitivity. Empirically, MARVL significantly outperforms existing VLM-reward methods on the Meta-World benchmark, demonstrating superior sample efficiency and robustness on sparse-reward manipulation tasks.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Reward Shaping, Vision-Language Models, Robotic Manipulation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_55GQOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

MARVL improves vision-language rewards for robotic reinforcement learning through scene–view disentanglement, stage-specific task-direction projection, and threshold-based reward shaping. It achieves strong results on eight Meta-World tasks, with additional, limited evidence of robustness across environments, camera settings, and RL algorithms.

Critical strengths
  1. Important problem and clear motivation. The framework systematically addresses viewpoint sensitivity, unreliable progress estimation, and semantic misalignment in VLM rewards.

  2. Strong main empirical results. MARVL achieves high final success rates across all eight Meta-World tasks and substantially outperforms the selected baselines on several tasks.

  3. Multiple forms of validation. Ablations and experiments involving camera changes, Panda-Gym, TD3, and generated subgoals provide initial support for the framework’s effectiveness.

Critical weaknesses
  1. Monotonicity claims exceed the evidence. The analysis relies on assumed progress monotonicity, additional alignment conditions, and a vanishing-noise limit, without adequately covering the practical setting of (α=0.8)(\alpha=0.8). Figure 17 also shows clear reward decreases.

  2. Additional supervision confounds attribution. MARVL uses paired multi-view data and manually specified intermediate goal images, but lacks simple goal-image reward and standard fine-tuning baselines under matched supervision.

  3. Novelty and related-work positioning are insufficiently established. The components closely relate to existing disentanglement, projection, and thresholding methods, while directly relevant work such as DrS [1], REDS [2], and T²-VLM [3] is omitted.

[1] Mu et al. DrS: Learning Reusable Dense Rewards for Multi-Stage Tasks. ICLR 2024.

[2] Kim et al. Subtask-Aware Visual Reward Learning from Segmented Demonstrations. ICLR 2025.

[3] Zhao et al. Training-free Generation of Temporally Consistent Rewards from VLMs. arXiv:2507.04789.

Questions for authors
  1. Can the authors establish monotonicity conditions for �0��0� and evaluate them using ranking correlations and monotonicity violation rates across multiple trajectories?

  2. Do all baselines receive the same pretrained models, adaptation data, and intermediate-goal information? How do simple image-distance rewards and existing projection methods perform under matched supervision?

  3. What is MARVL’s distinct contribution beyond existing view disentanglement, directional projection, and stage-aware rewards? Can the authors add the relevant discussion and comparisons?

  4. Table 2 lists ViT-B/32 together with a patch size of 14. Which configuration was actually used?

  5. Please specify the number of evaluation episodes, the meaning of shaded regions, and the number of seeds used in all supplementary experiments.

1 comment

Thank you for the clarifications. Your response addresses most of my concerns. I've increase my rating to weak acceptance.

Reviewer_vXaqOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes to use VLM / foundation model embeddings as rewards for RL agents. Critically the paper identifies shortcomings of existing VLM embeddings in this context, and as the key contribution, proposes three task-oriented fine-tuning techniques to improve VLM features. Results on meta-world are strong, although needing access to privileged information raises questions.

Critical strengths
  1. Well motivated and highly relevant problem space
  2. The method seems sound
  3. The analysis of failure modes of existing VLMs is insightful and interesting to the community
  4. The paper is well written, main results are nicely separated from supporting results in the appendix
  5. The results show clear improvement with MARVL
Critical weaknesses
  1. MARVL has access to additional privileged information that the baselines do not use, and that may render the tasks significantly easier. Most concerning is the “oracle” policy that is used to generate intermediate target images for the “direction” shaping objective, and the human labeled natural language descriptions of the intermediate steps. This leaks the oracle solution to the algorithm, indirectly reducing the RL exploration difficulty. Additionally the multi-view consistency objective also requires accessing the scene from multiple views, which is generally not possible in most real-world / realistic RL tasks.
  2. The objectives, in particular the progress objective feels like a heuristic, a stronger formal motivation / proof would make the paper stronger

Addressing the weaknesses in the rebuttal I would be willing to raise my score to weak accept

---- AFTER REBUTTAL: Thank you for the detailed response and new results! My concerns are addressed. Will raise my score to weak accept.

Questions for authors
  • FuRL results seem worse then the results published in the FuRL paper, what is the reason for that?
2 comments

Thank you for the detailed response and new results! My concerns are addressed. Will raise my score to weak accept.

Thank you very much for your encouraging response and for reconsidering your evaluation. We sincerely appreciate your thoughtful feedback and the time you took to examine our revisions and additional experiments. Your suggestions have been particularly valuable in helping us strengthen the paper's theoretical motivation and experimental validation.

Reviewer_7xauOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MARVL, a VLM-based reward-learning framework for reinforcement learning in robotic manipulation. The paper identifies three limitations of prior VLM-based rewards: weak spatial grounding, lack of progress awareness, and semantic misalignment. MARVL addresses these problems using Scene-View Decomposition, Multi-Stage Decomposition with Task Direction Projection, and Confidence-Thresholded Shaping, respectively, aiming to produce progress-aware and viewpoint-invariant dense rewards with noise suppressed. The paper benchmarks MARVL against baselines including FuRL, LIV, and Relay, on eight Meta-World tasks. Finally, ablation studies evaluate the impact of each module. Additional experiments evaluate transfer to Panda-Gym and generalization across RL backbones.

Critical strengths
  1. The paper demonstrates the explicit issues of existing VLM-based rewards and provides strong motivation for the proposed framework, with each component designed to address an identified limitation.
  2. The ablation study is comprehensive. It ablates the contribution of each proposed component and provides convincing empirical support for the overall framework design.
Critical weaknesses
  1. Confidence-Thresholded Shaping (CTS) sets the threshold θ\theta to be the m$_{th}$ quantile in the initial 10k steps of random exploration. However, the distribution of VLM scores might change as the policy and the state distributions evolve.
  2. Although the experiments include MetaWorld and Panda-Gym, the evaluated tasks show limited visual complexity and task-irrelevant visual distractions.
Questions for authors
  1. How do we ensure that the threshold in CTS estimated from the first 10k steps is sufficient and generalizes as training goes on?
  2. Could the authors compare MARVL against BIMI and VLM-RMs-GB that suppress irrelevant noise, as mentioned in related work, or against more recent baselines like Robometer and ReWind?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Author Response — Thank You to the Reviewers and Area Chair

We would like to sincerely thank all three reviewers and the Area Chair for their time, thoughtful feedback, and careful consideration of our work throughout the review and discussion process.

We are especially grateful for the constructive and open-minded discussions. The reviewers' questions challenged us to examine MARVL more rigorously, clarify its contributions, and strengthen the evidence supporting our claims. In response, we substantially revised the manuscript and added Appendices G–J (Figures 21–34), including:

  • Stronger theoretical foundations: A finite-alpha analysis with explicit conditional monotonicity guarantees and a clearer explanation of our design principles.
  • More comprehensive empirical validation: Matched-supervision baselines, 1M-step online RL evaluations, visual-distraction robustness tests, and additional reward diagnostics.
  • Improved reproducibility and positioning: Official-configuration baseline checks, expanded comparisons with related work, and clearer statistical reporting and methodological limitations.

Beyond addressing individual concerns, we believe these discussions have helped us present a more rigorous, transparent, and complete account of MARVL's contributions.

We particularly appreciate the reviewers' willingness to engage with our additional evidence and reconsider their initial assessments. Their careful evaluations and constructive suggestions have been invaluable in improving this work.

Finally, we thank the Area Chair for overseeing the evaluation process and considering the discussion as a whole.

Regardless of the final decision, we are genuinely grateful for the opportunity to improve MARVL through this exchange. Thank you again for your time, expertise, and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.