acceptodds
Under review as a conference paper at ICLR 2027

DRAVS: Directional Visual Reward Learning with Adaptive View Selection

Abstract

Vision-language models provide visual supervision for reinforcement learning, reducing reliance on manually designed rewards. Multi-view videos further alleviate occlusion by revealing task-relevant motion and contacts that may be hidden from a single viewpoint. Existing video-supervised state-wise reward formulations directly add learned state scores to task rewards. However, policies trained with such feedback can exhibit late-stage performance degradation, losing part of their earlier gains. Because the immediate visual term depends only on the current state, it does not explicitly distinguish progress from regression across transitions. Moreover, uniform random sampling of supervision views treats cameras equally despite differences in the visibility of task-relevant behavior. To address these limitations, we propose Directional Visual Reward Learning with Adaptive View Selection (DRAVS). First, Direction-Aware Visual Reward replaces state-wise addition with consecutive-state score differences computed using a scorer held fixed within each reinforcement-learning phase, providing direct feedback on progress and regression without additional vision-language model queries. Second, Budgeted Relative View Selection balances exploitation and exploration through sparse cross-view comparisons to adaptively select supervision views, with at most 5% additional scored clips over one-view-per-segment supervision. Compared with the state-of-the-art Multi-view Video Reward method, DRAVS raises mean final success rate on the ten-task Continual World benchmark built on MetaWorld from 0.54 to 0.69 and mean final return across nine HumanoidBench tasks from 616.9 to 836.3. Learning curves further show that DRAVS mitigates late-stage performance degradation and better preserves earlier gains. Additional qualitative visual comparisons are available at the anonymous project page: https://anon-visual-research.github.io/DRAVS/.

Primary area:
reinforcement learning
Keywords:
Reward Shaping, Reinforcement Learning, VLM Reward

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Rsm1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

DRAVS improves MVR's visual reward pipeline for reinforcement learning through two changes. Direction-Aware Visual Reward (DAVR) replaces an additive state score with the difference between consecutive state scores, keeping the relevance model and reference distribution frozen within each learning phase. Budgeted Relative View Selection (BRVS) learns task-level camera preferences from sparse comparisons of two views of the same execution segment The method retains environment rewards and state-based policy inputs.

Critical strengths
  1. DRAVS makes visual reward learning more useful on difficult control tasks, reported CW10 success increases from 0.54 to 0.69, while previously solved manipulation tasks remain solved (Table 1). This is a meaningful improvement to an existing reward-learning pipeline.

  2. Comparing two views of the same execution holds behavior fixed while choosing where to obtain visual supervision. BRVS uses this information without scoring every camera, bounding extra scored clips at 5%; this is useful when visual-model calls are expensive (Section 3.2; Eq. 9).

Critical weaknesses
  1. BRVS selects views with higher video–text scores for the same execution (Section 3.2). Such a view may make both successful and failed behavior look good, rather than help distinguish them. BRVS improves downstream results, but the paper does not directly show that its preferred views give more accurate success/failure rankings.

  2. The matched experiments cover only Hard-5 manipulation tasks (Section 4.3; Appendix D). On Sit_Hard, DRAVS also reuses each initial scene for 15 consecutive visual-supervision episodes, without clarifying whether MVR used the same schedule (Appendix C). These differences make it harder to judge how much of the humanoid-control gain comes from DRAVS itself.

Questions for authors
  1. Do the selected cameras distinguish successful from failed executions more accurately than the other cameras?

  2. Did MVR use the same 15-episode scene-reuse schedule on Sit_Hard? Can you compare MVR and DRAVS with the same code and scene-sampling settings on HumanoidBench?

Reviewer_9dT3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DRAVS modifies MVR's visual-reward pipeline through two components. Direction-Aware Visual Reward (DAVR) adds consecutive-state score differences to the environment reward, using a scorer and reference snapshot frozen within each learning phase. Budgeted Relative View Selection (BRVS) learns task-level camera preferences through sparse comparisons of the same execution from two viewpoints, then selects views for scorer supervision. Only the primary clip enters scorer training, and the comparison budget limits additional scored clips to 5%. Evaluation covers ten MetaWorld tasks and nine HumanoidBench tasks, with matched component and mechanism ablations on Hard-5.

Critical strengths
  1. This paper shows coherent intervention with substantial observed gains. By decoupling supervision view selection from reward conversion, the method substantially improves CW10 and H1, with matched Hard-5 ablations confirming each component's independent contribution.
  2. This paper provides thorough controls and theoretical clarity. Appendix D tests successor-state indexing, initial variance matching, discount rescaling, comparison budget, same-sequence pairing, and view-allocation choices. Appendix B carefully distinguishes its discounted-objective relation from preservation of the original task's optimal policy. Together with the scorer-freezing and replay details in Appendix C, these checks make the contribution more convincing.
Critical weaknesses
  1. All baseline results in the main tables 1 and 2 are directly imported from MVR paper. And all other empirical results are only reported on the Hard-5 subset of CW10. The close reproduced/published Hard-5 means, 0.25/0.24, are reassuring, but H1 protocol alignment remains important. Appendix C describes reusing each Sit_Hard initial scene for 15 visual-supervision episodes without establishing whether the published comparator used this exact schedule. So this raises a comparability quesiton.
  2. Appendix B relates DAVR to a rescaled SAVR augmented objective under fixed-scorer and boundary assumptions, making the empirical learning-dynamics evidence important. Table 4 reports DAVR at 0.34, versus 0.29 for SD-matched SAVR and 0.28 for discount-rescaled SAVR, over five seeds. The controls are appropriate, but marginal standard deviations do not establish the precision of these paired differences.
Questions for authors
  1. Which settings exactly match the published MVR protocol, including Sit_Hard scene reuse? Are existing matched H1 results or configuration comparisons available to resolve the remaining comparability question?
  2. Could the paired seed-level differences for the principal Table 4 contrasts be provided? Consistency across the matched runs would strengthen the interpretation of DAVR's gain beyond the tested calibrations.
Reviewer_yD4hOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Directional Visual Reward Learning with Adaptive View Selection (DRAVS), a framework for learning rewards. The approach includes two components: Direction-Aware Visual Reward (DAVR), which replaces the current visual state score with the score difference between consecutive states for reward calculation, and Budgeted Relative View Selection (BRVS), which selects camera views based on VLM alignment scores. The experiments on MetaWorld and HumanoidBench demonstrate the proposed approach's strong performance compared to existing baselines.

Critical strengths
  1. The approach demonstrates strong performance on both MetaWorld CW10 and HumanoidBench H1, outperforming or matching the baselines across all evaluated tasks. The results show the effectiveness of the overall approach.
  2. The ablation study is extensive, examining the contributions of DAVR, BRVS, reward-assignment calibration, and view selection mechanisms.
Critical weaknesses
  1. The paper proposes Budgeted Relative View Selection (BRVS) to select camera views based on the VLM alignment score, but how does the alignment score relate to how well the camera views discriminate between successful and unsuccessful behaviors?
  2. The motivation of Direction-Aware Visual Reward (DAVR) is that state-wise visual rewards do not explicitly distinguish progress from regression and show the late-stage degradation phenomenon. However, standard RL already accounts for future rewards in the Bellman equation. Could the authors clarify why the lack of immediate directional feedback results in the degradation, or how they are connected?
  3. The baselines VLM-RM and RoboCLIP are relatively outdated. More recent works, such as Robometer and ReWiND, are worth being compared against. Moreover, it is unclear whether the baselines use selected camera views or a single fixed view. Since the adaptive camera view selection is a key contribution, a fairer comparison would be to evaluate each baseline using each of the available camera views, since the baselines, except for MVR, are not designed for multiple camera views.
Questions for authors
  1. Could the authors clarify why camera preferences learned from higher VLM alignment scores are expected to correlate with better behavioral discrimination in reward learning? For example, as motivated in the introduction, do cameras preferred by BRVS actually provide better visibility of task-relevant motion and contacts than other viewpoints? It would be helpful if there could be qualitative comparisons of selected camera views or quantitative analysis of selected camera views' ability to distinguish behaviors while non-selected ones fail to do so.
  2. Refer to the second weakness.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank all reviewers for their constructive feedback. The reviewers recognized the substantial performance gains on both MetaWorld CW10 and HumanoidBench H1 (yD4h, 9dT3, Rsm1), the extensive ablations and careful controls (yD4h, 9dT3), and the value of budgeted same-execution view comparisons (Rsm1). Their main concerns focused on further validating the proposed mechanisms, strengthening baseline comparability and statistical evidence, and clarifying the presentation. We have added the corresponding experiments and analyses and revised the manuscript accordingly. Revisions are highlighted in blue in the revised PDF.

The key revisions are summarized below:

  1. Behavioral discrimination of BRVS-preferred views (yD4h, Rsm1; App. G, Tab. 8, Fig. 7): On held-out successful and failed segments rendered from all four cameras, BRVS-preferred views raise AUROC from 0.70 to 0.85 and ranking accuracy from 0.65 to 0.78, roughly doubling the success–failure score gap. Four-camera visual comparisons show that these views expose task-relevant cues such as block–target relations and foot–board contact.

  2. Bellman backups and late-stage degradation (yD4h; Sec. 3.3, Eq. 13; App. H, Tabs. 9–10): We clarified DAVR's relation to rescaled SAVR plus potential-based shaping. Empirically, as SAVR's success declines, the share of failed episodes receiving high visual scores rises from 18% to 43%, whereas it stays stable under DAVR (17% → 16%). Lowering the SAVR coefficient reduces degradation only at the cost of peak success, while DAVR retains both (degradation 0.01 vs. 0.09).

  3. Comparisons with recent reward models (yD4h; App. I, Tab. 11): We added Robometer and ReWiND, training every method with each camera fixed under a common TQC protocol. Even with BRVS disabled, DAVR achieves the highest Hard-5 success both averaged over cameras (0.33 vs. 0.28 for Robometer) and on each method's best camera (0.37 vs. 0.33).

  4. MVR comparability and results beyond Hard-5 (9dT3, Rsm1; Apps. J–K, Tabs. 12–13): Same-code MVR runs on four HumanoidBench tasks closely reproduce the published results, and DRAVS retains large gains on Stair and both Balance tasks. Under the same Sit_Hard scene-reuse schedule, MVR's return changes by only 6.8, so scene reuse does not account for the HumanoidBench gains. Extending the 2×2 component ablation to push-wall and peg-unplug-side, each component improves success and their combination performs best on both tasks.

  5. Paired statistical analysis (9dT3; App. D.4, Tab. 7): Paired seed-level analysis gives positive effects for DAVR (+0.11, 95% CI [0.05, 0.17]) and BRVS (+0.08, [0.04, 0.12]), and DRAVS exceeds matched MVR on all five seeds (+0.19, [0.09, 0.29]).

  6. Clearer method and protocol presentation (Rsm1; Secs. 3 and 4.1, Tab. 2): We clarified the flow from camera selection to state scoring and reward construction, explicitly stated the policy inputs, and documented the baseline setup and scene-sampling settings.

Detailed point-by-point responses are provided in the corresponding reviewer discussions. We welcome any follow-up questions.

AuthorsOpenReview ↗

Looking forward to hearing from you!

Dear Reviewers,

Thank you again for your detailed review. As the discussion period is drawing to a close, we would like to gently follow up on our responses to your concerns. We look forward to your further feedback.

Best regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.