acceptodds
Under review as a conference paper at ICLR 2027

Information-Guided Reward Learning for Contact-Rich Robotic Manipulation

Abstract

Contact-rich robotic manipulation requires distinguishing valid progress from ineffective motion. Yet correct insertion, misalignment, and jamming can appear visually similar, making image-based rewards unreliable. We propose an information-guided reward learning framework, which combines a physically supervised visual reward model with an action-history-based risk estimator. Privilege physical information, such as pose and contact signals, provide physical labels for reward learning, and do not appear during deployment. A dual-view visual model learns to predict positive, negative, or unclear physical progress, assigning unclear targets to visually ambiguous segments. The risk estimator learns whether visual positives are physically negative, based on executed-action history and visual class probabilities. At deployment, the risk estimator calibrate the visual reward model by changing high-risk positives to unclear. In Isaac Factory PegInsert, the framework suppresses 83.3% of negative-to-positive errors while retaining 82.9% of correct positives. Adding the learned reward to PPO increases aggregate success from 9.72% to 13.49%. On real Franka PegInsert data, Bayes-derived supervision increases mean episode-level student–teacher agreement from 43.70% to 59.32%, indicating improved supervision learnability rather than demonstrated gains in task success. These results support combining physical supervision and action history for more reliable contact-rich rewards.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Contact-rich manipulation, Bayesian contact belief estimation, Information-guided reward learning, Privileged physical supervision

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3W8ZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies reward learning for contact-rich manipulation, where correct insertion and misalignment may look nearly identical in images. It proposes a Bayesian-privileged reward learning framework that trains a dual-view visual reward model with privileged pose and contact labels. A risk estimator is applied to detect high-risk visual positives. The experiments show that the framework suppresses most negative-to-positive errors while retaining correct positives. Adding the learned reward to PPO also raises aggregate success.

Critical strengths
  • The paper addresses a concrete failure mode of image-based rewards in contact-rich tasks. Utilizing privileged physical informationis is well motivated and convincing.
  • The proposed history-conditioned risk estimator is a practical complement to the visual model. It uses executed action history, which is available at deployment, to selectively downgrade unreliable visual positives.
  • The evaluation connects reward quality to downstream deployment. It reports false-positive suppression and positive retention on Factory tasks, a matched PPO comparison, and student-teacher agreement on real Franka Peg Insert trajectories.
Critical weaknesses
  • The paper lacks substantial comparison with other methods. Although ablation studies are included, there is no comparison against other visual reward models or against methods that also use privileged physical information. As a result, the advantage of the Bayesian-privileged reward learning framework is not sufficiently demonstrated.
  • The gain of downstream policy is small, and absolute performance remains low. With no reported performance of other reward models under the same setting, it remains unclear how much the proposed framework could help downstream policy learning.
Questions for authors
  • Could the authors compare the framework with representative visual reward models and with methods that also train on privileged physical information, under the same Factory tasks and the same PPO setup?
  • The matched PPO success increases from 9.7% to 13.5%. Is this difference consistent across random seeds? If most negative-to-positive errors are already suppressed, why does aggregate success stay near 13%?
Reviewer_VccxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to give robots reliable rewards for contact-rich manipulation. In tasks such as peg insertion, correct insertion and jamming can look almost the same in camera images, so a visual model may reward physically wrong actions. The authors use physical information available during training, such as object pose and contact state, to label whether a short interaction makes progress, fails, or remains unclear.

They train a visual reward model on these labels and use executed action history to check whether a positive visual prediction is likely to be a mistake; risky positives are changed to unclear rather than rewarded. The method is evaluated on simulated assembly tasks, downstream PPO on PegInsert, and offline real-robot Franka data. The results mainly show that action history can reduce false positive rewards, while the real-robot evidence measures label learnability rather than improved task success.

Critical strengths
  1. S1. The paper addresses a practical reward problem. In contact-rich tasks, correct and incorrect contact can look similar in images, so a visual reward may reinforce physically wrong actions. Using physical information during training and action history to check positive predictions is an intuitive way to reduce this problem.

  2. S2. The action-history selector shows promising results. At the same false-positive suppression rate, it retains 82.9% of valid positives versus 52.9% for RGB-only selection. The learned reward also improves PegInsert PPO success from 9.72% to 13.49%, providing initial evidence that it can help policy learning.

Critical weaknesses
  1. W1. The implementation of Φ\Phi is unclear. I could not find a description of how the paper determines whether a visual window contains sufficient evidence to preserve a physical label or convert it to U. Since this rule determines the visual training targets, the authors should clarify how it is implemented.

  2. W2. Evaluation is limited to a small number of tasks. The reward model is evaluated on three simulated tasks, but the selector, downstream PPO, and real-robot experiments are all limited to PegInsert. The method's effectiveness on other contact-rich tasks therefore remains unknown.

Questions for authors
  1. Implementation of Φ\Phi . How does the method determine whether a visual window contains enough evidence to preserve a physical label or assign U?
  2. Task generality. Can the selector and downstream PPO be evaluated on other tasks?
  3. Possible reward cycling. Since P gives +1 and N gives −0.2, could an approach–retreat sequence receive net positive reward without real task progress? How does the method prevent this behavior?
Reviewer_WBT9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces an information-guided reward learning framework to enable robot distinguish valid progress from ineffective motion. It leverages physical information such as pose and contact signals to construct physical reward labels. Beyond reward learning, the paper also introduces risk estimator to calibrate reward model by changing high-risk positives to unclear.

Experiments show that the Bayes-based reward model (auther's method) outperforms SFT on peginsert task in simulation. No realworld experiment on policy learning, only one task for evaluating the performance of reward model.

Critical strengths

1.Good idea. The author recognizes the importance of physical information for policy learning and attempts to incorporate such information through the reward model. This direction is commendable. 2. The design of the reward model is sound, and the experiments are relatively comprehensive. The manuscript provides a detailed description of the proposed reward model and includes a comprehensive set of experiments—covering three simulated tasks, one real-world robot task, and an ablation study.

Critical weaknesses

1.Few experiment. The manuscript only evaluates the reward model on three tasks in simulation and one task in realworld. And these tasks are not contact-rich in commmon sense, for example, peg insertion. Adding more experiments—particularly contact-rich tasks, such as wiping a whiteboard—would enhance the credibility of the evaluation. 2. Vague experimental setup. While the authors provide a detailed description of the reward model and the risk estimator, the paper lacks a specific explanation of how the rewards learned by the reward model are utilized for downstream policy learning. 3. Lack of ablation. The paper does not conduct an ablation study on the "risk estimator"—a key module—within the context of downstream policy learning; consequently, readers are left unable to discern the role this module plays in policy learning.

Questions for authors

1.How exactly is the reward learned by the reward model utilized during model training? Does it replace the original reward used in PPO, or is there some other mechanism at play? 2. What impact does a risk estimator have on policy learning?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.