- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies reward repair in reinforcement learning using sparse numerical feedback on trajectory outcomes. The proposed method combines feedback on partial trajectories with exploration data to learn a reward correction and a potential function. It calibrates and validates the correction before keeping it fixed during subsequent policy training. The paper also provides a conditional regret analysis and reports improved recovery on continuous-control tasks and reward-hacking benchmarks.
Critical strengths
-
The theoretical analysis provides a coherent basis for the method. It distinguishes the role of intermediate outcomes from that of broader behavioral coverage, motivating the combination of partial-trajectory feedback and exploration data. The conditional regret analysis also helps clarify the assumptions under which the repaired reward can support policy improvement.
-
The ablations offer relevant support for the main design choices. Replacing exploration trajectories with equally many source-policy trajectories weakens recovery, and removing the potential function also reduces performance. These results support the roles of behavioral coverage and potential estimation. The main experiments also show recovery of intended task behavior under different reward misspecifications.
Critical weaknesses
-
The overall feedback and interaction costs remain unclear. The equal-trajectory coverage ablation and the comparison with HERON using all stored repair feedback provide useful evidence for the method's design. However, the reported comparisons do not clearly account for the additional feedback and interactions used for exploration, calibration, validation, and optional refitting. A complete accounting would make it easier to assess the gains relative to the supervision and interaction required by each method.
-
The empirical behavior of the selection rule remains unclear when no candidate passes validation. The method explicitly selects an active correction in this case and retains its failed validation status; the conditional theoretical analysis also covers this selection branch. However, the experiments do not report how often it is used or how intended return changes in those cases. These results would help assess the practical reliability of deploying the selected correction, since the uncorrected reward is used as a validation reference but is excluded from selection.
Questions for authors
-
Beyond the reported feedback controls, could the authors clarify the additional feedback and environment interactions required for exploration, calibration, validation, and optional refitting, and how these costs are accounted for in the baseline comparisons?
-
How often does no candidate pass validation, and how does deploying the selected correction affect intended return in those cases?
Thank you for the detailed responses. The results when no candidate passes validation address my main questions. Overall, my concerns are generally addressed, and I will keep my score.