acceptodds
Under review as a conference paper at ICLR 2027

Bellman Calibration for V-Learning in Offline Reinforcement Learning

Abstract

Calibration is a standard tool for assessing and correcting the reliability of predictions in supervised learning, but is less developed for long-horizon value prediction in reinforcement learning, where the prediction target is not directly observed. We extend calibration to the Bellman fixed-point setting by introducing Bellman calibration, which requires the Bellman equation to hold conditionally on the predicted value. Equivalently, among states assigned the same predicted value, the average one-step Bellman target should equal that prediction. Because this target itself depends on the value predictor, calibration becomes a fixed-point problem. We define a calibration error that quantifies systematic Bellman residuals conditional on the predicted value and develop doubly robust estimators from off-policy data. We establish a calibration–refinement bound that relates calibration error to value-prediction error, separating the component that post-hoc calibration can correct from information lost by the original predictor. We then develop Fitted Bellman Calibration, a model-agnostic post-hoc procedure that iteratively recalibrates a learned value predictor using one-dimensional histogram or isotonic regression, without refitting the underlying high-dimensional value function. We establish finite-sample guarantees showing one-dimensional nonparametric calibration rates for histogram and isotonic calibration, together with value-error guarantees for fixed-partition calibration, without imposing realizability or Bellman-completeness assumptions on the calibrated predictor class.

Primary area:
reinforcement learning
Keywords:
offline reinforcement learning, value function estimation, Bellman calibration, off-policy evaluation, model calibration, fitted value iteration

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_4HyyOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Bellman calibration for value functions in offline reinforcement learning, requiring predictions to match the average Bellman target among states assigned the same predicted value. It introduces Fitted Bellman Calibration, which iteratively learns a scalar transformation of an existing value predictor using histogram or isotonic regression. The paper develops doubly robust estimators of calibration error and finite sample guarantees, and establishes a bound relating value prediction error to calibration error and the information lost by the original predictor.

Critical strengths
  • The paper introduces Bellman calibration as a reliability criterion that can be estimated from offline transition data. The bound relating calibration to value error clarifies which errors can be addressed by recalibration and which arise from information missing in the original predictions.
  • The proposed procedure learns only a scalar transformation of an existing value predictor, making it straightforward to apply to different models. The histogram analysis provides calibration and value error bounds without requiring realizability or Bellman completeness of the original value function class.
Critical weaknesses
  • Improvements in Bellman calibration do not consistently translate into more accurate policy value estimates, particularly after longer FQE training in Table S2. This leaves it unclear when calibration should be applied if the practical goal is accurate policy evaluation.

  • The connection between the theoretical algorithm and the implementation used in the synthetic experiments needs clarification. Appendix B.1 uses weighted regression with clipped importance weights, but the finite sample consequences of these changes relative to Algorithm 1 are not explicitly analyzed.

Questions for authors
  • In Table S2, the HalfCheetah med policy after 7,000 FQE updates has a CAL ratio of 0.07 but an OPE error ratio of 5.41. Appendix C computes CAL under the logged distribution, while the policy value estimate uses initial states and actions from the target policy. Could the authors investigate how much of this discrepancy comes from distribution mismatch and how much comes from refinement error? In particular, would weighting calibration by the discounted occupancy of the target policy, as suggested in the conclusion, improve OPE accuracy in these cases?

  • The discussion following Theorem 3 states that the finite iteration term ∥T^πVK−T^πVK−1∥2,ρ\|\widehat T_\pi V_K-\widehat T_\pi V_{K-1}\|_{2,\rho} decays geometrically, referring to Appendix J.4. However, Lemma S12 bounds successive iterates by a geometric term plus statistical and auxiliary estimation terms that do not vanish as KK increases. What additional conditions ensure geometric decay for the fitted iteration? Can this term be made negligible without introducing an additional dependence on 1/(1−γ)1/(1-\gamma) into the calibration error bound?

  • Algorithm 1 regresses doubly robust pseudo outcomes on the original prediction, whereas Appendix B.1 uses weighted least squares with importance weights clipped at 20. Even when q^v=0\widehat q_v=0, the histogram updates differ at finite sample sizes. Algorithm 1 averages weighted targets within each bin, while weighted least squares divides their sum by the total weight in that bin. Could the authors clarify which guarantees apply to the implemented update and how clipping affects its calibration target? A comparison with the update in Algorithm 1, together with sensitivity to the clipping threshold, would help establish whether these implementation choices materially affect the results.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 4Hyy (Part 1 of 2)

Revised manuscript (anonymous PDF; key changes in blue): PDF

Thank you for your positive assessment of Bellman calibration and the calibration--refinement bound. Since submission, we have added Hopper and DeepOPE experiments using both offline and on-policy calibration data, revised Algorithm 1 and the main finite-sample analysis to match the importance-weighted regression used in our experiments, and proved geometric convergence of the fitted iteration.

1. Calibration data and policy-value accuracy (Critical Weakness 1; Question 1).

The HalfCheetah example illustrates why improving calibration under the logged-data distribution need not improve policy-value accuracy under the target policy. Our value-error bounds include coverage factors for transferring calibration guarantees between these distributions, which can be large when the logged data poorly cover the target-policy occupancy. We suspect distribution mismatch explains much of the discrepancy, although Table S2 does not let us quantify its contribution relative to refinement error. Isotonic calibration chooses its plateaus adaptively rather than imposing a fixed coarse partition, avoiding one potential source of information loss, but it cannot recover information missing from the original predictor.

To investigate the role of coverage, we added experiments on five Hopper transition kernels and four DeepOPE environments. We improve target-policy coverage by mixing offline trajectories from a policy library that includes the target policies. Averaging over target policies and seeds, calibration of predictors trained for 7,000 FQE updates reduces initial-state value RMSE by 52--80% and absolute policy-value error by 60--83% relative to uncalibrated predictors across all nine benchmarks.

We also calibrate the same pretrained predictors using 3,000 new on-policy transitions. Calibration reduces initial-state value RMSE by 74--97% and absolute policy-value error by 83--98%. We compare against three alternative uses of the same on-policy batch: retraining FQE from scratch on the pooled offline and on-policy data, fine-tuning the pretrained predictor, and training FQE from scratch on the on-policy data alone. Calibration achieves lower average error on both metrics than all three alternatives across all nine benchmarks. These results support the coverage explanation.

As you suggest, and as we discuss in the conclusion, discounted-occupancy weighting offers another way to align calibration with the target policy without requiring on-policy data. With exact weights, calibration error under the target-policy discounted occupancy directly controls scalar policy-value bias. Our strong results with on-policy calibration suggest that occupancy weighting could also improve policy-value accuracy in cases like the HalfCheetah example. Investigating whether these gains extend to estimated occupancy weights, while accounting for ratio-estimation error and weight variability, is a promising direction for future work.

AuthorsOpenReview ↗

Response to Reviewer 4Hyy (Part 2 of 2)

2. Convergence of the fitted iteration (Question 2).

Thank you for catching this. The bound in Lemma S12 was too loose to justify the geometric-decay statement. In the revision, we give a direct contraction argument for the fitted updates. Conditional on the calibration observations and weights, both the histogram and isotonic updates are γ\gamma-contractions in the supremum norm, so

\le \gamma^{K-1}\\|V_1-V_0\\|_\infty.$$ Thus, no additional statistical conditions are needed for geometric decay. We have revised Theorems 3 and 5 accordingly, replacing the output-dependent iteration term by $2CM\gamma^K/(1-\gamma)$ (Appendices G.4 and H). This term can be made negligible by increasing $K$, and the factor $1/(1-\gamma)$ appears only in this geometrically decaying remainder, not in the leading statistical calibration error. **3. Weighted regression and clipping (Critical Weakness 2; Question 3).** In our experiments, we use importance ratios as regression weights rather than to construct unbiased pseudo-outcomes. As you point out, these updates differ in finite samples: within each histogram bin, weighted regression normalizes by the sum of importance weights, whereas the pseudo-outcome update averages weighted targets over observations. With exact weights, however, both have the same population target. We have therefore revised Algorithm 1 to use importance-weighted regression, and our main finite-sample guarantees now cover this update directly. We moved the more general pseudo-outcome construction to Appendix E, since it also accommodates model-based and doubly robust constructions. Our guarantees also allow clipped estimated weights. Any discrepancy between the clipped weights and the true ratio contributes to the weight-estimation error in our bounds. If the true ratios lie below the clipping threshold everywhere, clipping does not change the limiting calibration target; otherwise, it can introduce bias. We also compared the two regression updates across all six synthetic settings and found similar improvements over raw FQE. To assess sensitivity to clipping, we reran the pseudo-outcome and doubly robust variants without clipping. Across 100 paired replications per setting, removing the clipping threshold of 20 changes either value-error metric by less than 0.4% in every setting.

Thank you for the detailed response and substantial revisions, which address my concerns about convergence and the consistency between the theoretical analysis and implementation. The additional experiments strengthen the empirical evidence, although it remains unclear when calibration reliably improves policy value estimates under limited coverage. Nevertheless, as my concerns have been largely addressed, I will raise my rating to 3.

Reviewer_JwdKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Bellman calibration, extending supervised learning calibration to Bellman fixed points in offline reinforcement learning. It proposes Fitted Bellman calibration with theoretical guarantees and evaluates the method on synthetic and D4RL benchmarks. Experiments show substantial improvements in calibration, but mixed effects on policy-value prediction accuracy.

Critical strengths
  1. The paper formulates Bellman calibration as a fixed-point problem. The calibration–refinement bound separates systematic Bellman miscalibration from information lost by the original predictor, clarifying both the potential and the limitations of post-hoc correction.
  2. The method adjusts only the scalar output without refitting the underlying value predictor. Its doubly robust target recovers the appropriate conditional Bellman target.
Critical weaknesses
  1. Uncertain downstream benefit: Large CAL improvements can accompany substantially worse value estimates: HalfCheetah-medium shows a CAL ratio of 0.07 but an OPE-error ratio of 5.41. A practical criterion for when calibration is beneficial remains missing.
  2. There is a potentially consequential proof gap. Lemma S7 appears to incorrectly infer ∥πh∥n,S≤∥h∥n,S,A\|\pi h\|_{n,S}\leq\|h\|_{n,S,A} from Jensen’s inequality: target-policy action averages are not controlled by the observed-action empirical norm alone. This leaves an important step supporting the finite-sample guarantees unresolved.
Questions for authors
  1. The paper identifies FBC as fitted value iteration restricted to functions of the original prediction. Which technical result goes beyond applying existing state-aggregation analysis to this scalar representation?
  2. Equation (1) implies that even the constant predictor c=Eρ,π[R]/(1−γ)c=\mathbb E_{\rho,\pi}[R]/(1-\gamma) is perfectly Bellman calibrated, how much of the observed improvement comes from correcting predictions versus pooling away meaningful distinctions?
  3. Justification of Lemma S7.
2 comments
AuthorsOpenReview ↗

Response to Reviewer JwdK

Revised manuscript (anonymous PDF; key changes in blue): PDF

1. Downstream value accuracy (Weakness 1).

Our theory explicitly distinguishes calibration from downstream value accuracy. Calibration error is measured under the calibration-data distribution, while target-policy value error also depends on coverage of the target-policy distribution and the remaining refinement error. The HalfCheetah example illustrates this distinction: calibration improves value estimation for the less-trained FQE predictor but worsens it for the better-trained predictor, even though calibration error decreases in both cases. Thus, reducing calibration error alone need not improve OPE when the calibration data poorly cover the target-policy occupancy. An important practical consideration is therefore target-policy coverage, rather than calibration error alone.

To investigate this, we added experiments with improved target-policy coverage across nine new Hopper and DeepOPE benchmarks. First, we modified the offline data mixture so that target-policy trajectories make up about 10% of the data. With this improved overlap, calibration reduces absolute policy-value error by 60--83% across all nine benchmarks. Second, we use 3,000 new on-policy transitions to calibrate the same pretrained offline predictors. Calibration reduces policy-value error by 83--98% and outperforms three alternative uses of the same additional data: retraining FQE on the pooled data, fine-tuning the existing predictor, and training FQE from scratch on the on-policy data alone. These results demonstrate the potential of calibration when a small amount of high-quality data is available to align the predictor with the target policy. Under the target-policy discounted occupancy distribution, calibration error also directly controls scalar policy-value bias.

2. Proof of Lemma S7 (Weakness 2; Question 3).

Thank you for catching this. We agree that the empirical-norm inequality does not follow directly from Jensen's inequality. We have corrected the proof by constructing a common cover under an equal mixture of the state and next-state distributions, which controls both norms simultaneously without relying on the invalid comparison. The corrected entropy argument establishes the same finite-sample rates, so all our main theoretical guarantees remain unchanged.

3. Beyond state aggregation (Question 1).

Our main contribution is to formalize Bellman calibration for offline RL and study its relationship to value accuracy, how calibration error can be estimated, and how calibrated predictors can be constructed with finite-sample guarantees. In particular, our calibration--refinement bound distinguishes calibration error, which post-hoc calibration can reduce, from refinement error, which captures information not retained by the original predictor. We also develop doubly robust calibration-error estimators and establish finite-sample guarantees for histogram and isotonic calibration.

State aggregation enters only one part of this analysis. For histogram calibration, discretizing the original predictions induces a state aggregation, and we use standard aggregation arguments to derive the corresponding value-error bound. The broader calibration framework, calibration-error estimators, and finite-sample analyses of our calibration procedures go beyond applying existing state-aggregation results to a scalar representation.

4. Constant predictors and loss of information (Question 2).

Yes, a constant predictor can be perfectly Bellman calibrated while discarding nearly all information needed for accurate value prediction. This illustrates why calibration alone is insufficient for value accuracy. Our calibration--refinement decomposition captures this explicitly: calibration error measures the systematic discrepancy that calibration can correct, while refinement error captures the predictive information lost by pooling distinct states.

Our calibration procedures aim to improve calibration without unnecessarily discarding information from the original predictor. For histogram calibration, excessive pooling can increase refinement error, so we recommend choosing the number of bins by cross-validation rather than fixing an overly coarse partition. Our finite-sample guarantees also accommodate this data-adaptive choice when the selected number of bins is bounded with high probability by a deterministic sequence, as discussed in the paper.

Isotonic calibration avoids choosing the number of bins altogether. It fits a monotone transformation of the original predictions, preserving their ordering. The identity map is also contained in the calibration class, so an already well-calibrated predictor need not be changed.

Thank you for the detailed response and the additional experiments. The new proof of Lemma S7 resolves the mathematical issue. Also, the clarification of downstream value accuracy helps address my concerns about uncertain downstream benefit. Moreover, I appreciate the explanations of state aggregation and information loss, which answer my remaining questions. There is no more question for me. I appreciate the authors’ efforts and will maintain my rating of 3.

Reviewer_kY7WOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper extends the notion of calibration to value functions in reinforcement learning. By taking conditional expectation of the bootstrapped value function conditioned on the current state value (instead of the current state), the paper introduces the notion of Bellman calibration and Bellman calibration error (which is weaker than Bellman fixed-point condition and Bellman error). The paper establishes an upper bound on the value prediction error in terms of the Bellman calibration error, a coverage assumption, and the best value-prediction error based on one-dimensional transformation of the original predicted value function. The paper also proposed several methods for post-hoc calibration of value functions, including methods based on histogram binning and isotonic regression.

Critical strengths

Calibration is an important concept in prediction and decision-making, and extending it to value functions in reinforcement learning is a novel and valuable contribution. The paper also presents several new theoretical results about value calibration error.

Critical weaknesses
  • In Theorem 3 and 5, the bound contains a term ∣T^πVK−T^πVK−1∣2,ρ|\widehat{\mathcal T}_\pi V_K-\widehat{\mathcal T}_\pi V_{K-1}|_{2,\rho}. This term is said to converge at a rate γK\gamma^K, but this is not formally justified in the paper. With the right-hand-side term depending on the algorithm's output, the current bound is not very complete. It would make the result cleaner by concretely proving the exponential convergence of this term.
  • No theoretical results are provided for the doubly-robust estimator.
Questions for authors

Please address the questions in the weakness section. Additionally, the convergence rates in the paper are usually slower than the 1/sqrt{n} rate that is typical in policy evaluation itself (e.g. Theorem 5 has a n^{-1/3} rate), while I understand that this is coming from the binning techniques, it is not clear whether this is unavoidable. In particular, it looks a bit weird to me, as Bellman calibration is a weaker requirement than Bellman error, while a \sqrt{n} rate is possible for the latter.

2 comments
AuthorsOpenReview ↗

Response to Reviewer kY7W

Revised manuscript (anonymous PDF; key changes in blue): PDF

Thank you for recognizing the contribution of extending calibration from supervised learning to value prediction in offline RL, as well as the theoretical guarantees we establish.

1. Convergence of the iteration error (Weakness 1; Theorems 3 and 5).

Thank you for pointing this out. In the revision, we eliminate the output-dependent finite-iteration term from the main bounds and replace it with the explicit geometric-decaying remainder 2CMγK/(1−γ)2CM\gamma^K/(1-\gamma). We obtain this by first expressing the iteration error as the empirical L2L^2 distance between consecutive fitted iterates evaluated at the observed next states, rather than its population counterpart. This quantity is observable and vanishes when the empirical iterates stabilize. We then prove that, with the calibration observations and weights fixed, the importance-weighted fitted update is a γ\gamma-contraction in the supremum norm, so the iteration error decays geometrically.

The factor 1/(1−γ)1/(1-\gamma) appears only in this geometrically decaying remainder and does not enter the leading statistical calibration term. Thus, the effective horizon affects how many fitted iterations are needed to make the iteration error negligible, but does not worsen the statistical calibration rate.

2. Theory for double robustness (Weakness 2).

We agree that a more complete theoretical analysis of the doubly robust estimator of squared calibration error in Section 2.4 was warranted. In the revision, we establish double robustness, root-nn consistency, and asymptotic normality for this cross-fitted estimator in Appendix E.2, extending the framework of Xu and Yadlowsky (2022) to our RL setting.

Double robustness also plays a separate role in our estimation procedure. Theorem 2 establishes that the conditional bias of our Bellman pseudo-outcome is a product of the importance-weight and Bellman-target regression errors. This product remainder explicitly appears in our finite-sample guarantees for calibration error and value prediction accuracy.

3. Convergence rates (Question).

The n−1/3n^{-1/3} rate is not intrinsic to Bellman calibration. It is important to distinguish estimating the squared calibration error of a fixed value function from constructing a value function with vanishing calibration error.

The former is a scalar estimation problem, for which our doubly robust estimator achieves a root-nn rate under the conditions of Appendix E.2. This is analogous to estimating scalar policy values or mean squared Bellman errors.

By contrast, constructing a value function with vanishing calibration error requires learning a calibration function through nonparametric regression of Bellman targets on the original value predictions. This is a one-dimensional nonparametric learning problem, explaining the n−1/3n^{-1/3} rate for isotonic regression and the bin-dependent rates for histogram regression. Crucially, these rates are independent of the state-space dimension because Bellman calibration only requires conditioning on the scalar value prediction. In contrast, constructing a Bellman-consistent value function over the full state space generally suffers from the curse of dimensionality, with potentially much slower rates.

Moreover, if a value estimator already achieves a root-nn rate for its L2L^2 Bellman residual, then its Bellman calibration error under the same distribution also has a root-nn rate by Jensen's inequality. Our analysis does not require such a guarantee.

References.

The paper addressed my main concern. I have increased my score accordingly.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions

Our paper introduces Bellman calibration: among states assigned the same value prediction, the average one-step Bellman target should equal that prediction. We develop a calibration–refinement bound, doubly robust calibration-error estimators, and histogram and isotonic methods that recalibrate an existing predictor without refitting it.

We have expanded the experiments and theory to address the reviewers’ main concerns:

  1. Benefits for value prediction. We added experiments across five Hopper transition kernels and four DeepOPE environments with controlled target-policy coverage. At 7,000 FQE updates, offline calibration reduces initial-state value RMSE by 52–80% across these nine settings. With the same predictors, calibration using 3,000 additional on-policy transitions reduces RMSE by 74–97% and gives lower RMSE than retraining on pooled data, fine-tuning, or training on the additional batch alone. Each method receives the same additional data. We also report scalar policy-value error and paired uncertainty estimates, directly examining the downstream quantity reviewers asked about.

  2. Theory for the implemented algorithm. We revised Algorithm 1 and the finite-sample analysis to cover the importance-weighted regression used in the experiments, including estimated and clipped weights.

  3. Stronger theoretical justification. We prove geometric convergence of the fitted updates and give an explicit bound on the remaining iteration error. We correct the covering-number argument identified by Reviewer JwdK while preserving the stated rates, and add double-robustness and asymptotic-normality results for the calibration-error estimator.

These revisions strengthen the theoretical guarantees and provide a clearer picture of when calibration improves value prediction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.