- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
When fine-tuning vision-language-action (VLA) policies using reinforcement learning (PPO) against learned visual rewards, reward overoptimization frequently occurs. Learned progress rewards often assign high scores to near-goal failures (e.g., hovering an object directly over a target without releasing the gripper), causing proxy reward values to increase while actual task success drops sharply (e.g., a 9.9 percentage point drop in ManiSkill). The authors propose a diagnostic framework based on exponential tilting of initial-policy trajectories to establish a reference distribution. By evaluating a checkpoint’s matched-proxy residual (actual task success minus reachable success by reweighting initial behavior at the same proxy score mean), they formally quantify whether degradation stems from reward overoptimization.
Proposed Framework (LSAR): Label-Selective Audit and Refresh (LSAR) dynamically manages gold simulator predicate queries under a strict overall label budget:
(1) Audit Trigger: At designated checkpoints, LSAR sequentially queries gold labels for a random sample of the top-20% highest-scoring recent rollouts. It computes a Beta posterior on their precision and triggers a refresh only when the probability of precision falling below an initial reference exceeds a defined threshold.
(2) Active Acquisition & Update: Upon triggering, it acquires informative unlabeled rollouts ranked by a rank-product metric (high score, reward ensemble disagreement, and feature novelty). An auxiliary success head is updated using all accumulated gold labels to correct representation distortion.
Key Findings & Results:
(1) Under a 1,000-label budget, LSAR achieves 72.3% final success on ManiSkill (spending 765 labels on average) and 40.8% on LIBERO-Long (spending 637 labels).
(2) It consistently outperforms fixed-reward policies (41.4% / 24.9%) as well as periodic top-score labeling (66.9% / 33.4%) and cost-matched controls, spending labels only when genuine degradation occurs.
Critical strengths
(1) The authors clearly identify an Under-Explored Failure Mode and propose a method on Principled, Budget-Aware Dynamic Refresh (LSAR).
(2) Experiments in simulation environment are extensive and demonstrate the effectiveness of the proposed method.
Critical weaknesses
(1) Most of the experiments are in simulation environment. The reviewer is concerned about how it would perform in real-world scenario.
Questions for authors
Please refer to the weakness part.