acceptodds
Withdrawn from ICLR 2027

Auditing Learned Robot Rewards

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Learned visual rewards can score near-goal failures highly, allowing PPO to increase proxy reward while task success falls. Fixed-P8 runs show a 9.9 pp ManiSkill peak-to-final success decline and a matched-proxy residual that turns negative. Label-Selective Audit and Refresh (LSAR) uses random gold audits of top-scored trajectories and refreshes the reward when their precision falls below a reference level. At a 1,000-label cap, LSAR ends at 72.3% on ManiSkill and 40.8% on LIBERO-Long, versus 66.9% and 33.4% for periodic top-score labeling, using a mean of 765 and 637 method-requested gold labels, respectively; shared setup and diagnostic labels are excluded. Shadow precision, held-out audits, matched-cost controls, and component analyses characterize the refresh mechanism and its label cost.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robot reinforcement learning, learned reward models, reward auditing, reward hacking, label-efficient learning, robotic manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Ai1KOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

When fine-tuning vision-language-action (VLA) policies using reinforcement learning (PPO) against learned visual rewards, reward overoptimization frequently occurs. Learned progress rewards often assign high scores to near-goal failures (e.g., hovering an object directly over a target without releasing the gripper), causing proxy reward values to increase while actual task success drops sharply (e.g., a 9.9 percentage point drop in ManiSkill). The authors propose a diagnostic framework based on exponential tilting of initial-policy trajectories to establish a reference distribution. By evaluating a checkpoint’s matched-proxy residual (actual task success minus reachable success by reweighting initial behavior at the same proxy score mean), they formally quantify whether degradation stems from reward overoptimization.

Proposed Framework (LSAR): Label-Selective Audit and Refresh (LSAR) dynamically manages gold simulator predicate queries under a strict overall label budget:

(1) Audit Trigger: At designated checkpoints, LSAR sequentially queries gold labels for a random sample of the top-20% highest-scoring recent rollouts. It computes a Beta posterior on their precision and triggers a refresh only when the probability of precision falling below an initial reference exceeds a defined threshold.

(2) Active Acquisition & Update: Upon triggering, it acquires informative unlabeled rollouts ranked by a rank-product metric (high score, reward ensemble disagreement, and feature novelty). An auxiliary success head is updated using all accumulated gold labels to correct representation distortion.

Key Findings & Results:

(1) Under a 1,000-label budget, LSAR achieves 72.3% final success on ManiSkill (spending 765 labels on average) and 40.8% on LIBERO-Long (spending 637 labels).

(2) It consistently outperforms fixed-reward policies (41.4% / 24.9%) as well as periodic top-score labeling (66.9% / 33.4%) and cost-matched controls, spending labels only when genuine degradation occurs.

Critical strengths

(1) The authors clearly identify an Under-Explored Failure Mode and propose a method on Principled, Budget-Aware Dynamic Refresh (LSAR).

(2) Experiments in simulation environment are extensive and demonstrate the effectiveness of the proposed method.

Critical weaknesses

(1) Most of the experiments are in simulation environment. The reviewer is concerned about how it would perform in real-world scenario.

Questions for authors

Please refer to the weakness part.

Reviewer_etQeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates reward overoptimization in robotic reinforcement learning, where policies optimized against learned visual rewards may achieve higher predicted rewards while actual task success deteriorates. The authors introduce a matched-proxy selection reference to analyze this divergence and propose Label-Selective Audit and Refresh (LSAR), which audits high-reward trajectories using ground-truth success labels and selectively updates the reward model when its precision degrades.

Critical strengths

Reward overoptimization is a meaningful challenge in robot RL, particularly when policies are trained using learned visual rewards rather than ground-truth task success.

Critical weaknesses
  • Heavy reliance on ground-truth success labels limits real-world applicability. My main concern is that the proposed audit mechanism fundamentally depends on privileged ground-truth success labels provided by the simulator. While the paper motivates the difficulty of learning reliable visual rewards, its solution assumes access to an external oracle that can determine whether high-reward trajectories actually succeed. In real-world robotics, obtaining such labels may itself be a challenging problem, particularly for complex, long-horizon manipulation tasks involving partial observability, ambiguous task completion, or subtle physical interactions.
  • Limited experimental diversity and lack of real-world validation. The experiments are conducted entirely in simulation, with one ManiSkill placement task and ten LIBERO-Long tasks. However, learned rewards may fail for substantially different reasons, including occlusion, incorrect physical-state estimation, contact dynamics, partial task completion, and complex multi-stage dependencies. It remains unclear whether the proposed audit criterion generalizes across these different failure modes. Moreover, the reward-refresh experiments focus on the progress-based P8 reward model, without demonstrating whether the method generalizes to other reward formulations, including the success-based S8 model considered in the paper. More diverse experiments, particularly on physical robots, would substantially strengthen the conclusions.
Questions for authors

N/A

Reviewer_7poUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces LSAR, a label-budgeted reward-model refresh approach that uses sequential random audits of top-scoring samples to determine when additional labels and reward updates are needed. The proposed framework includes a matched-proxy selection diagnostic and a decomposition of success changes into reference and residual components. The method is evaluated on ManiSkill, where the authors report a final success rate of 72.3% while using an average of 765 labels from a budget of 1,000. The experiments also compare audit-triggered refresh with scheduled refresh strategies under a shared labeling budget.

Critical strengths
  1. The paper considers the relevant problem of managing labeling budgets for reward-model updates and proposes an audit-based mechanism for determining when additional labels are needed.
Critical weaknesses
  1. The paper suffers from substantial presentation and clarity issues that make it difficult to understand the proposed contribution. The abstract does not clearly establish the research problem or explain the proposed method. Important terminology, including P8, S8, and progress reward, is introduced without sufficient definition, explanation, or citation. The paper appears to assume familiarity with these concepts throughout the discussion. Furthermore, the presentation frequently emphasizes technical details, numerical results, and implementation-specific terminology before establishing the broader motivation and research objective. Similar issues appear in the conclusion, making the paper read more like a technical report than a clearly structured research contribution. There are also formatting issues, including merged lines around line 123.
  2. The novelty and significance of the proposed approach are not sufficiently established. It is unclear whether LSAR represents a new reward-modeling algorithm, an optimization procedure, or a heuristic for deciding when to update an existing reward model. The relationship between the proposed diagnostic, audit mechanism, and reward refresh procedure is not adequately motivated in terms of the central research contribution. Although empirical results are reported, the paper does not clearly explain what generalizable methodological insight or improvement the proposed approach establishes. Consequently, it is difficult to assess the significance of the reported findings.
Questions for authors
  1. What is the primary methodological contribution of LSAR, and how does it differ from existing reward-model update or label-selection procedures?
  2. What do P8, S8, and progress reward represent, and how are these concepts defined and used in the proposed framework?
  3. How do the matched-proxy diagnostic and sequential audit mechanism contribute to the proposed method, and which components are essential to its effectiveness?
  4. What is the main generalizable finding of the empirical evaluation beyond the reported success rates and labeling budget utilization?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.