acceptodds
Under review as a conference paper at ICLR 2027

: Repairing Only What Misleads the Policy from Sparse Outcomes

Abstract

Dense rewards guide learning well but are often misspecified, leading to reward hacking. Sparse trajectory outcomes are reliable but provide weak guidance. Existing methods either learn a new reward from feedback or restrict how far the policy departs from reference behavior. We take a different route and repair the dense proxy itself, using sparse outcomes to correct the errors that cause hacking while keeping its guidance. The difficulty is that an outcome only measures the total reward of a trajectory, so a correction can fit every outcome and still favor the wrong policy. We quantify this ambiguity with an ambiguity modulus and use it to bound the regret of the repaired policy. The analysis shows that the ambiguity arises in two ways: errors can cancel out within a trajectory, or lie on behavior the data never covers. Fitting the outcomes more accurately fixes neither. We therefore propose Adaptive Identification for Reward Repair (), which queries outcomes on trajectory prefixes to catch errors that cancel, and collects outcomes from an exploration policy to cover what the data missed. also estimates a potential function alongside the correction, so that harmless shaping in the proxy is not mistaken for error. The final correction is frozen and passed to a standard RL learner. On 22 continuous control tasks with constructed reward misspecification, instantiated with BAC achieves the highest aggregate normalized recovery score among the evaluated methods. We further apply with PPO to three published benchmarks and observe substantial improvements over the baseline methods in both experimental settings.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Reward Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_FYhwOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies reward repair in reinforcement learning using sparse numerical feedback on trajectory outcomes. The proposed method combines feedback on partial trajectories with exploration data to learn a reward correction and a potential function. It calibrates and validates the correction before keeping it fixed during subsequent policy training. The paper also provides a conditional regret analysis and reports improved recovery on continuous-control tasks and reward-hacking benchmarks.

Critical strengths
  1. The theoretical analysis provides a coherent basis for the method. It distinguishes the role of intermediate outcomes from that of broader behavioral coverage, motivating the combination of partial-trajectory feedback and exploration data. The conditional regret analysis also helps clarify the assumptions under which the repaired reward can support policy improvement.

  2. The ablations offer relevant support for the main design choices. Replacing exploration trajectories with equally many source-policy trajectories weakens recovery, and removing the potential function also reduces performance. These results support the roles of behavioral coverage and potential estimation. The main experiments also show recovery of intended task behavior under different reward misspecifications.

Critical weaknesses
  1. The overall feedback and interaction costs remain unclear. The equal-trajectory coverage ablation and the comparison with HERON using all stored repair feedback provide useful evidence for the method's design. However, the reported comparisons do not clearly account for the additional feedback and interactions used for exploration, calibration, validation, and optional refitting. A complete accounting would make it easier to assess the gains relative to the supervision and interaction required by each method.

  2. The empirical behavior of the selection rule remains unclear when no candidate passes validation. The method explicitly selects an active correction in this case and retains its failed validation status; the conditional theoretical analysis also covers this selection branch. However, the experiments do not report how often it is used or how intended return changes in those cases. These results would help assess the practical reliability of deploying the selected correction, since the uncorrected reward is used as a validation reference but is excluded from selection.

Questions for authors
  1. Beyond the reported feedback controls, could the authors clarify the additional feedback and environment interactions required for exploration, calibration, validation, and optional refitting, and how these costs are accounted for in the baseline comparisons?

  2. How often does no candidate pass validation, and how does deploying the selected correction affect intended return in those cases?

1 comment

Thank you for the detailed responses. The results when no candidate passes validation address my main questions. Overall, my concerns are generally addressed, and I will keep my score.

Reviewer_2vTZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies reward repair: correcting a misspecified dense proxy reward using trusted trajectory outcomes, while keeping the proxy's useful shaping. The authors bound regret with an "ambiguity modulus" and identify two sources of ambiguity:

  1. Temporal cancellation, where errors at different steps cancel within a trajectory.
  2. Missing coverage, where errors lie on behavior the data never visits.

Based on this, they propose AIR^2, which learns an additive correction to the proxy from these outcomes, using outcomes on trajectory prefixes and from an exploration policy to address the two sources of ambiguity. Experiments on 22 constructed control tasks and three published benchmarks report the highest aggregate scores among the compared methods.

Critical strengths
  1. The framing is nice. Asking which policy comparisons the outcomes can identify, rather than trying to recover the reward itself, seems like the right target for reward repair. The cited literature and benchmarks seem appropriate too.
  2. Several of the recoveries in Fig. 3 are large (e.g., Door-open, Peg-unplug), and the reward misspecifications are documented in enough detail to reproduce (Table 2, App. F).
Critical weaknesses

The weaknesses I would most like to see addressed are:

  1. AIR^2 seems to receive richer feedback than the baselines. AIR^2 gets cardinal returns at multiple horizons from paired continuations launched from restored states (line 1427). The preference-based baselines learn from preferences (line 375), and ORPO and Max-Min receive no outcome feedback (line 1953). To be fair, the comparison with HERON given all stored repair feedback (lines 466–468) partly addresses this. However, HERON still only sees preferences, so I found it hard to tell how much of the gain comes from the method rather than the feedback.

    • Relatedly, the conclusion claims "substantial improvements in both prevention and recovery" (lines 484–485), but in recovery AIR^2 has the highest mean only on Tomato (Table 3).
  2. Proposition 1 (lines 167–170) seems perhaps a tad trivial and doesn't add much to the main body. Theorem 2 (lines 188–189) reads more like a remark than a full theorem. I think this space would be better spent on the regret bound (Theorem 3).

  3. Some of the results in Fig. 3, Fig. 5 and Table 3 are noisy. For example, several Fig. 3 panels overlap heavily with the best baseline (e.g., Walker, Ant, Pendulum), and most of the "no potential" error bars in Fig. 5 cross zero. It is also not stated what the ±\pm and shaded regions represent (lines 1626–1632).

Minor suggestions:

  • Add a limitations section. The requirements (state resets, an evaluator that returns partial returns) are only mentioned in passing (lines 130–131, 1427).
  • In Figure 2, the "Action intervention" branch isn't explained in the text.
  • In Figure 4 (Traffic prevention), the zoom inset covers much of the main panel. Consider moving it outside the axes.
Questions for authors
  1. Appendix G.2 gives pair counts only for the temporal ablation. How many outcome queries does AIR^2 use in the main experiments, how many environment steps go into training the exploration policy and collecting the paired continuations, and how were the preference labels for the preference-based baselines generated? Apologies if I've missed this elsewhere in the text.
  2. Are the outcomes in the experiments noisy? If not, did you test how robust AIR^2 is to outcome noise?
Reviewer_Q4M2OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how sparse, trusted numerical trajectory feedback can address misleading rewards. The authors decompose the proxy reward into the intended reward, potential-based shaping, and residual bias. Experiments cover 22 constructed continuous-control tasks and three published benchmarks, using BAC and PPO to evaluate recovery and prevention, demonstrating the effectiveness of the proposed method.

Critical strengths
  1. Section 3 and Appendix A–D carefully distinguish between fitting trajectory outcomes and identifying correct policy comparisons. Moreover,Equations (2) and (7) retain the potential contributions at nonterminal endpoints, while Theorem 3 distinguishes fitting, representation, deployment, and optimization errors. These distinctions are consequential because they explain why accurate reward prediction alone is insufficient to guarantee good control performance.

  2. The experimental evaluation is broad and thorough. It covers manipulation, locomotion, and other control tasks, includes two policy learners, and uses six random seeds per method. These results support the practical value of the method.

Critical weaknesses
  1. The empirical evaluation does not sufficiently isolate the proposed mechanisms from confounding factors, and an important gap remains in the specification of the constructed rewards. The multi-horizon ablation also increases the number of feedback queries, as the authors explicitly acknowledge. It is therefore hard to independently establish that querying trajectory prefixes is better than querying more complete trajectories under the same budget.

  2. Furthermore, Appendix F.1 states that a “common potential” is added to the proxy reward, but does not provide its precise definition, magnitude, or implementation. Table 2 describes only the reward components before this potential is added. This omission prevents full reproduction of the actual proxy reward and makes it difficult to determine whether the potential-modeling ablation is primarily driven by the artificially added shaping.

Questions for authors
  1. Could the authors provide the the numbers of trajectory pairs used for fitting, calibration, validation, and refitting? Are the additional data collection and exploration costs included in the horizontal axes?

  2. What are the expression, magnitude, and termination/truncation handling of the common potential mentioned in F.1? Does modeling the potential still provide benefits when this additional potential is not introduced?

  3. Are there comparisons against simplified residual regression or return decomposition under the same numerical supervision and total query budget, as well as comparisons between querying multiple prefixes and querying more complete trajectories? These results would directly affect the assessment of the core contribution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.