acceptodds
Under review as a conference paper at ICLR 2027

Calibrate Before You Learn: Coverage-Aware Rewards for Offline RL

Abstract

Offline reinforcement learning has access to trajectory outcomes before training begins, but early temporal-difference updates still rely on a critic that has not learned to predict them. We propose coverage-aware reward calibration to use these outcomes as additional supervision. Before critic training, we fit a shared state potential to return-to-go labels on a nearest-neighbor graph, with local geometry controlling each label's influence. During training, we add the potential difference to the reward, amplify the original reward component, and gradually reduce both changes while retaining the base learner's update rules. Under the stated terminal convention, potential-based shaping preserves optimal action ordering at fixed coefficients. We relate potential error to behavior-advantage information and characterize when calibrated weights improve on uniform smoothing within locally constant-value regions. A two-step analytical example connects this improvement to the first Bellman update. Experiments cover sparse navigation, dexterous manipulation, and continuous control. Matched ablations show gains beyond reward scaling alone and support graph smoothing and geometry-based weighting, with additional controls across offline learners. These results support using precomputed outcomes to assist offline TD learning.

Primary area:
reinforcement learning
Keywords:
offline RL

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_77r2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduced a reward calibration method for offline reinforcement learning. The idea is to use compute a coverage-aware reward signal derived from offline episode returns which is adjusted over training helping algorithms to overcome detrimental Bellman updates initially.

Critical strengths
  • From a mathematical perspective, the method seems to be transparently laid out and it is compatible, in principle, with any offline RL algorithm.
  • First promising indicators on standard offline RL datasets when compared with other offline RL methods are given
Critical weaknesses
  • I think the major weakness of the paper is the type of experiments conducted. To me, it sees that work here is half done. The paper is about a reward calibration method that should be agnostic to the base learner. Nevertheless, in the experiments, it is only run in combination with TD3-BC and only compared with vanilla offline RL algorithms that do not use any calibration. Why not combine it with CQL or IQL as well? I think there have to be experiments showing the effect of the method on more algorithms. Moreover, it must be benchmarked against other reward calibration methods. In its current form, the experiments in 5.1 are, at best, very early signals that something has some promising direction than a robust evaluation. Even with these promising signals, the experiments currently do not enough to prove that the proposed method has the right formulation carying the success. Structural ablation studies on the key design choices are missing For instance: How does data quality, data size, or episode length affect the method? How do hyperparameters of the offline RL algorithms affect performance? Can the proposed calibration compensate bad hyperparameters of the vanilla algorithm?
  • The paper presents a very special construction of a reward calibration. Beside the fact I find central parts of the construction hard to understand, the construction comes largely without motivation, intuition, or theoretic gurantees why it is supposed to works. The authors do not enough to communicate the larger picture or give inuition why the design of their method is how it is. To name one example: Why are the weights wi,jw_{i,j} computed using the very special form of U(x)U(x) given in Equation 5?
Questions for authors

See my questions in the weakness section.

Reviewer_L7KdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a coverage-aware reward calibration method for offline reinforcement learning, which aims to address the issues that conventional temporal-difference learning over-relies on inaccurate critic estimates in the early stage of training and fails to fully exploit the trajectory return information already available in offline data.

Critical strengths

(1) The method adopts a modular design that organically integrates coverage-aware weighting, graph-smoothed potential fitting, and dynamic reward calibration. The overall technical route is clear and offers good generality and scalability. (2) Theoretical analysis is developed from multiple perspectives, including policy invariance, the relationship between the potential function and the advantage function, and the bias and variance of graph smoothing, providing sufficient theoretical support for the method design. (3) Extensive evaluations are conducted on 28 D4RL datasets. Experiments on IQL and CQL also support the transferability of the method.

Critical weaknesses

(1) Geometry weighting is the core of the method, but the authors have not yet sufficiently verified the relationship between the local geometry score and the reliability of RTG labels, and there is a lack of experimental evidence demonstrating that the proposed geometry weights can accurately reflect label uncertainty. (2) The relationship between the potential function and the final performance improvement is unclear. For example, on the AntMaze datasets, the rank correlation between the potential function and RTG is very low, yet the method still achieves significant performance gains. The authors have not yet adequately explained this phenomenon. (3) The theoretical assumptions are relatively strong, and its applicability in practical scenarios still requires further discussion.

Questions for authors

Please refer to the weaknesses.

Reviewer_SgXcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes coverage-aware reward calibration to improve offline reinforcement learning using return information already available in the dataset.

The method first computes return-to-go labels from recorded trajectories and fits a state potential using a nearest-neighbor graph. The graph smooths the return labels, while geometry-based weights control how much each label influences the potential.

During training, the method uses potential-based reward shaping and reward scaling to provide additional supervision for TD learning. Both modifications gradually decrease as training progresses.

The authors provide theoretical analysis of policy preservation and the effects of potential estimation errors. Experiments on D4RL and V-D4RL demonstrate improvements across navigation, manipulation, and locomotion tasks. Ablations examine the contributions of reward scaling, graph smoothing, and geometry-based weighting.

Critical strengths
  1. A practical way to use observed returns for offline value learning. Offline datasets already contain information about future rewards, but standard TD learning must gradually propagate this information through bootstrapping. The proposed method uses these returns to construct a state potential that provides useful supervision from the beginning of training. It can also be combined with existing offline RL algorithms without changing their main learning objectives.

  2. Strong empirical performance with useful ablations. The method achieves substantial improvements over TD3+BC on several D4RL tasks, particularly on AntMaze and Adroit. The authors also provide matched ablations that separate reward scaling, return-to-go supervision, graph smoothing, and geometry-based weighting. Additional experiments with IQL and CQL show that the method can improve different offline RL algorithms. These results support the practical usefulness of the proposed approach.

Critical weaknesses
  1. It is unclear whether reward shaping offers advantages over directly learning from trajectory returns. The main motivation of the paper is that offline datasets already contain future return information that could help TD learning. However, a simpler approach would be to use these returns to pretrain or initialize the critic before standard offline RL training. Potential-based reward shaping is also closely related to value initialization, as acknowledged in the paper. The proposed method instead constructs a separate state potential and gradually modifies the reward during training. It is unclear whether this provides additional benefits over directly using the same return information to train the critic. The ablations show that reward scaling alone already accounts for a large part of the improvement on some tasks. For example, on AntMaze-medium-diverse, reward scaling improves the score from 61.4 to 92.1, while the full method reaches 94.7. Although the potential provides larger improvements on other tasks, a controlled comparison with return-based critic pretraining is needed to clarify its additional contribution.

  2. The reliability of graph smoothing under complex dynamics is unclear. The method uses a nearest-neighbor graph to share return information between nearby states. This assumes that states close in the chosen representation tend to have similar values. However, geometric proximity does not necessarily imply similar future returns. This is particularly important in contact-rich manipulation, where small differences in contact conditions or object motion can lead to very different outcomes. Smoothing return labels across such states could remove useful information rather than improve value estimation. Furthermore, the proposed geometry-based weights mainly reflect local data coverage. A region with many samples may still contain highly variable returns, especially when the dataset includes different behaviors or contact outcomes. Therefore, high coverage does not necessarily mean that the return labels are reliable. The theoretical analysis explains when weighted smoothing can improve over uniform smoothing, but it requires a relationship between the weights and label reliability that is not directly verified for the proposed geometry scores. The Adroit experiments are encouraging, but they do not directly test whether the graph preserves important return differences across contact modes. The improvements over uniform graph weighting are also relatively small, and the matched ablations do not report seed-level uncertainty.

Questions for authors
  1. Comparison with direct return supervision: Could the authors compare the proposed method against pretraining or initializing the critic using the same return-to-go information, followed by standard offline RL training? This would help clarify why reward shaping is preferable to directly using the available return labels.

  2. Graph smoothing under complex dynamics: How does the method handle nearby states with substantially different returns, particularly in contact-rich manipulation? Could the authors analyze return differences between graph neighbors or compare the current graph with a dynamics-aware representation? This would help establish when geometric proximity is a reliable basis for value smoothing.

  3. Reliability of coverage-based weighting: Do the proposed geometry scores correlate with actual return-label uncertainty or prediction error? Could the authors provide such an analysis and report seed-level uncertainty for the improvements over uniform graph weighting in Table 2? This would help determine whether the proposed weights reliably improve the potential estimate.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.