- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
DRAVS improves MVR's visual reward pipeline for reinforcement learning through two changes. Direction-Aware Visual Reward (DAVR) replaces an additive state score with the difference between consecutive state scores, keeping the relevance model and reference distribution frozen within each learning phase. Budgeted Relative View Selection (BRVS) learns task-level camera preferences from sparse comparisons of two views of the same execution segment The method retains environment rewards and state-based policy inputs.
Critical strengths
-
DRAVS makes visual reward learning more useful on difficult control tasks, reported CW10 success increases from 0.54 to 0.69, while previously solved manipulation tasks remain solved (Table 1). This is a meaningful improvement to an existing reward-learning pipeline.
-
Comparing two views of the same execution holds behavior fixed while choosing where to obtain visual supervision. BRVS uses this information without scoring every camera, bounding extra scored clips at 5%; this is useful when visual-model calls are expensive (Section 3.2; Eq. 9).
Critical weaknesses
-
BRVS selects views with higher video–text scores for the same execution (Section 3.2). Such a view may make both successful and failed behavior look good, rather than help distinguish them. BRVS improves downstream results, but the paper does not directly show that its preferred views give more accurate success/failure rankings.
-
The matched experiments cover only Hard-5 manipulation tasks (Section 4.3; Appendix D). On Sit_Hard, DRAVS also reuses each initial scene for 15 consecutive visual-supervision episodes, without clarifying whether MVR used the same schedule (Appendix C). These differences make it harder to judge how much of the humanoid-control gain comes from DRAVS itself.
Questions for authors
-
Do the selected cameras distinguish successful from failed executions more accurately than the other cameras?
-
Did MVR use the same 15-episode scene-reuse schedule on Sit_Hard? Can you compare MVR and DRAVS with the same code and scene-sampling settings on HumanoidBench?
General Response
We thank all reviewers for their constructive feedback. The reviewers recognized the substantial performance gains on both MetaWorld CW10 and HumanoidBench H1 (yD4h, 9dT3, Rsm1), the extensive ablations and careful controls (yD4h, 9dT3), and the value of budgeted same-execution view comparisons (Rsm1). Their main concerns focused on further validating the proposed mechanisms, strengthening baseline comparability and statistical evidence, and clarifying the presentation. We have added the corresponding experiments and analyses and revised the manuscript accordingly. Revisions are highlighted in blue in the revised PDF.
The key revisions are summarized below:
Behavioral discrimination of BRVS-preferred views (yD4h, Rsm1; App. G, Tab. 8, Fig. 7): On held-out successful and failed segments rendered from all four cameras, BRVS-preferred views raise AUROC from 0.70 to 0.85 and ranking accuracy from 0.65 to 0.78, roughly doubling the success–failure score gap. Four-camera visual comparisons show that these views expose task-relevant cues such as block–target relations and foot–board contact.
Bellman backups and late-stage degradation (yD4h; Sec. 3.3, Eq. 13; App. H, Tabs. 9–10): We clarified DAVR's relation to rescaled SAVR plus potential-based shaping. Empirically, as SAVR's success declines, the share of failed episodes receiving high visual scores rises from 18% to 43%, whereas it stays stable under DAVR (17% → 16%). Lowering the SAVR coefficient reduces degradation only at the cost of peak success, while DAVR retains both (degradation 0.01 vs. 0.09).
Comparisons with recent reward models (yD4h; App. I, Tab. 11): We added Robometer and ReWiND, training every method with each camera fixed under a common TQC protocol. Even with BRVS disabled, DAVR achieves the highest Hard-5 success both averaged over cameras (0.33 vs. 0.28 for Robometer) and on each method's best camera (0.37 vs. 0.33).
MVR comparability and results beyond Hard-5 (9dT3, Rsm1; Apps. J–K, Tabs. 12–13): Same-code MVR runs on four HumanoidBench tasks closely reproduce the published results, and DRAVS retains large gains on Stair and both Balance tasks. Under the same Sit_Hard scene-reuse schedule, MVR's return changes by only 6.8, so scene reuse does not account for the HumanoidBench gains. Extending the 2×2 component ablation to push-wall and peg-unplug-side, each component improves success and their combination performs best on both tasks.
Paired statistical analysis (9dT3; App. D.4, Tab. 7): Paired seed-level analysis gives positive effects for DAVR (+0.11, 95% CI [0.05, 0.17]) and BRVS (+0.08, [0.04, 0.12]), and DRAVS exceeds matched MVR on all five seeds (+0.19, [0.09, 0.29]).
Clearer method and protocol presentation (Rsm1; Secs. 3 and 4.1, Tab. 2): We clarified the flow from camera selection to state scoring and reward construction, explicitly stated the policy inputs, and documented the baseline setup and scene-sampling settings.
Detailed point-by-point responses are provided in the corresponding reviewer discussions. We welcome any follow-up questions.