- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Bellman calibration for value functions in offline reinforcement learning, requiring predictions to match the average Bellman target among states assigned the same predicted value. It introduces Fitted Bellman Calibration, which iteratively learns a scalar transformation of an existing value predictor using histogram or isotonic regression. The paper develops doubly robust estimators of calibration error and finite sample guarantees, and establishes a bound relating value prediction error to calibration error and the information lost by the original predictor.
Critical strengths
- The paper introduces Bellman calibration as a reliability criterion that can be estimated from offline transition data. The bound relating calibration to value error clarifies which errors can be addressed by recalibration and which arise from information missing in the original predictions.
- The proposed procedure learns only a scalar transformation of an existing value predictor, making it straightforward to apply to different models. The histogram analysis provides calibration and value error bounds without requiring realizability or Bellman completeness of the original value function class.
Critical weaknesses
-
Improvements in Bellman calibration do not consistently translate into more accurate policy value estimates, particularly after longer FQE training in Table S2. This leaves it unclear when calibration should be applied if the practical goal is accurate policy evaluation.
-
The connection between the theoretical algorithm and the implementation used in the synthetic experiments needs clarification. Appendix B.1 uses weighted regression with clipped importance weights, but the finite sample consequences of these changes relative to Algorithm 1 are not explicitly analyzed.
Questions for authors
-
In Table S2, the HalfCheetah med policy after 7,000 FQE updates has a CAL ratio of 0.07 but an OPE error ratio of 5.41. Appendix C computes CAL under the logged distribution, while the policy value estimate uses initial states and actions from the target policy. Could the authors investigate how much of this discrepancy comes from distribution mismatch and how much comes from refinement error? In particular, would weighting calibration by the discounted occupancy of the target policy, as suggested in the conclusion, improve OPE accuracy in these cases?
-
The discussion following Theorem 3 states that the finite iteration term decays geometrically, referring to Appendix J.4. However, Lemma S12 bounds successive iterates by a geometric term plus statistical and auxiliary estimation terms that do not vanish as increases. What additional conditions ensure geometric decay for the fitted iteration? Can this term be made negligible without introducing an additional dependence on into the calibration error bound?
-
Algorithm 1 regresses doubly robust pseudo outcomes on the original prediction, whereas Appendix B.1 uses weighted least squares with importance weights clipped at 20. Even when , the histogram updates differ at finite sample sizes. Algorithm 1 averages weighted targets within each bin, while weighted least squares divides their sum by the total weight in that bin. Could the authors clarify which guarantees apply to the implemented update and how clipping affects its calibration target? A comparison with the update in Algorithm 1, together with sensitivity to the clipping threshold, would help establish whether these implementation choices materially affect the results.
3 comments
Response to Reviewer 4Hyy (Part 2 of 2)
2. Convergence of the fitted iteration (Question 2).
Thank you for catching this. The bound in Lemma S12 was too loose to justify the geometric-decay statement. In the revision, we give a direct contraction argument for the fitted updates. Conditional on the calibration observations and weights, both the histogram and isotonic updates are -contractions in the supremum norm, so
\le \gamma^{K-1}\\|V_1-V_0\\|_\infty.$$ Thus, no additional statistical conditions are needed for geometric decay. We have revised Theorems 3 and 5 accordingly, replacing the output-dependent iteration term by $2CM\gamma^K/(1-\gamma)$ (Appendices G.4 and H). This term can be made negligible by increasing $K$, and the factor $1/(1-\gamma)$ appears only in this geometrically decaying remainder, not in the leading statistical calibration error. **3. Weighted regression and clipping (Critical Weakness 2; Question 3).** In our experiments, we use importance ratios as regression weights rather than to construct unbiased pseudo-outcomes. As you point out, these updates differ in finite samples: within each histogram bin, weighted regression normalizes by the sum of importance weights, whereas the pseudo-outcome update averages weighted targets over observations. With exact weights, however, both have the same population target. We have therefore revised Algorithm 1 to use importance-weighted regression, and our main finite-sample guarantees now cover this update directly. We moved the more general pseudo-outcome construction to Appendix E, since it also accommodates model-based and doubly robust constructions. Our guarantees also allow clipped estimated weights. Any discrepancy between the clipped weights and the true ratio contributes to the weight-estimation error in our bounds. If the true ratios lie below the clipping threshold everywhere, clipping does not change the limiting calibration target; otherwise, it can introduce bias. We also compared the two regression updates across all six synthetic settings and found similar improvements over raw FQE. To assess sensitivity to clipping, we reran the pseudo-outcome and doubly robust variants without clipping. Across 100 paired replications per setting, removing the clipping threshold of 20 changes either value-error metric by less than 0.4% in every setting.Thank you for the detailed response and substantial revisions, which address my concerns about convergence and the consistency between the theoretical analysis and implementation. The additional experiments strengthen the empirical evidence, although it remains unclear when calibration reliably improves policy value estimates under limited coverage. Nevertheless, as my concerns have been largely addressed, I will raise my rating to 3.
Response to Reviewer 4Hyy (Part 1 of 2)
Revised manuscript (anonymous PDF; key changes in blue): PDF
Thank you for your positive assessment of Bellman calibration and the calibration--refinement bound. Since submission, we have added Hopper and DeepOPE experiments using both offline and on-policy calibration data, revised Algorithm 1 and the main finite-sample analysis to match the importance-weighted regression used in our experiments, and proved geometric convergence of the fitted iteration.
1. Calibration data and policy-value accuracy (Critical Weakness 1; Question 1).
The HalfCheetah example illustrates why improving calibration under the logged-data distribution need not improve policy-value accuracy under the target policy. Our value-error bounds include coverage factors for transferring calibration guarantees between these distributions, which can be large when the logged data poorly cover the target-policy occupancy. We suspect distribution mismatch explains much of the discrepancy, although Table S2 does not let us quantify its contribution relative to refinement error. Isotonic calibration chooses its plateaus adaptively rather than imposing a fixed coarse partition, avoiding one potential source of information loss, but it cannot recover information missing from the original predictor.
To investigate the role of coverage, we added experiments on five Hopper transition kernels and four DeepOPE environments. We improve target-policy coverage by mixing offline trajectories from a policy library that includes the target policies. Averaging over target policies and seeds, calibration of predictors trained for 7,000 FQE updates reduces initial-state value RMSE by 52--80% and absolute policy-value error by 60--83% relative to uncalibrated predictors across all nine benchmarks.
We also calibrate the same pretrained predictors using 3,000 new on-policy transitions. Calibration reduces initial-state value RMSE by 74--97% and absolute policy-value error by 83--98%. We compare against three alternative uses of the same on-policy batch: retraining FQE from scratch on the pooled offline and on-policy data, fine-tuning the pretrained predictor, and training FQE from scratch on the on-policy data alone. Calibration achieves lower average error on both metrics than all three alternatives across all nine benchmarks. These results support the coverage explanation.
As you suggest, and as we discuss in the conclusion, discounted-occupancy weighting offers another way to align calibration with the target policy without requiring on-policy data. With exact weights, calibration error under the target-policy discounted occupancy directly controls scalar policy-value bias. Our strong results with on-policy calibration suggest that occupancy weighting could also improve policy-value accuracy in cases like the HalfCheetah example. Investigating whether these gains extend to estimated occupancy weights, while accounting for ratio-estimation error and weight variability, is a promising direction for future work.