acceptodds
Under review as a conference paper at ICLR 2027

Calibrated Pessimism in Offline Reinforcement Learning: Auditing Support-Conditioned Reliability and Control Transfer

Abstract

Offline reinforcement learning often relies on pessimism to reduce the influence of value estimates in weakly supported regions, yet the uncertainty signals used to construct pessimistic penalties are rarely tested as calibrated error surrogates. We study this reliability question directly in a focused five-seed HalfCheetah Medium case study. Using leakage-controlled train/calibration/validation splits, we calibrate global, uncertainty-ratio, and local-support-aware bounds for an observable held-out Bellman-residual proxy, and separately evaluate marginal coverage, support-conditioned reliability, and sharpness. Calibrated constructions attain near-nominal marginal coverage, but this aggregate view hides substantial redistribution of reliability across support regimes: at 90% nominal coverage, the global bound remains relatively narrow while under-covering low-support transitions, whereas adaptive scaling raises low-support coverage at the cost of substantially wider bounds and lower high-support coverage. We then reuse the frozen calibration objects in an evaluation-time candidate-reranking study with matched proposals, no retraining, and no return-based tuning. The reranking interface itself substantially underperforms the deterministic actor, and within this restricted interface we find no evidence that calibrated penalties improve return over unpenalized Q reranking. Our results therefore show, within the scope of this case study, that uncertainty magnitude, marginal statistical calibration, and downstream control utility can diverge substantially. The standard split-conformal guarantee applied here concerns only marginal coverage of the observable residual proxy under exchangeability assumptions; it is not a guarantee for true Bellman error, policy value, or return.

Primary area:
reinforcement learning
Keywords:
offline reinforcement learning, uncertainty calibration, pessimism, conformal prediction, uncertainty quantification

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_y2ibOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies whether uncertainty-based pessimism in offline reinforcement learning is statistically reliable and whether a calibrated uncertainty can improve action selection. They propose a held-out one-step Bellman-residual proxy and use a split-conformal procedure. Using HalfCheetah Medium, they evaluate the calibrated bounds as penalities and show that they do not improve downstream control. The main conclusion is that uncertainty magnitude, statistical calibration, and downstream control utility should be evaluated separately.

Critical strengths

This is a meaningful and practically relevant question by separating uncertainty magnitude, statistical calibration, and downstream control utility. They are often implicitly conflated in offline RL.

Critical weaknesses
  1. The empirical study is restricted to a single HalfCheetah Medium task with a IQL backbone and five training seeds. It is unclear whether the conclusions generalize across environments or learning algorithms.
  2. Tha main phenomenon is not particular novel. The central observation that marginal conformal coverage cannot guarantee subgroup-conditioned miscoverage. This is already a well-known limitation of marginal conformal validity.
  3. The downstream control experiment is difficult to interpret. The unpenalized Q-reranking interface itself performs much worse than the original deterministic actor(from 12115 to 3640). How to interpret the failure in policy control as a limitation of calibrated pessimism itself.
  4. The control experiment fixes the penalty weight at λ=1\lambda=1, even though the calibrated constructions have substantially different bound widths.
Questions for authors

See the weaknesses

Reviewer_UY7MOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper audits whether uncertainty signals used for pessimism in offline RL are calibrated error surrogates. On HalfCheetah Medium with a frozen IQL backbone and episode-disjoint splits, it calibrates global, ratio-based, and support-aware split-conformal bounds against a held-out one-step Bellman-residual proxy. All calibrated bounds reach near-nominal marginal coverage but allocate coverage and width very differently across low- and high-support regions. Reusing the bounds as penalties in evaluation-time action reranking yields no return improvement over unpenalized Q reranking.

Critical strengths

Careful and transparent protocol for offline RL methods.

Critical weaknesses

The study is too narrow: it covers a single task, dataset quality, and backbone. The findings also do not translate into design guidance.

Paper format does not follow standard ICLR template.

Questions for authors

IQL performs only in-support maximization and uses no uncertainty penalty, so the audited penalty plays no role in training. Do the findings hold with a backbone that trains with uncertainty pessimism?

How is the ensemble disagreement u(x) defined? With IQL’s two critics, it appears to be a very crude uncertainty estimate.

Rescaling raw disagreement is already absorbed by tuning the penalty weight λ in offline RL methods?

Reviewer_NH5HOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper conducts a case‑study on uncertainty‑driven pessimism in offline reinforcement learning. It adopts split‑conformal calibration to held out Bellman‑residual proxies and evaluates marginal coverage, support-conditioned reliability, and sharpness. These metrics show heterogeneous reliability across data‑support regimes at near‑nominal marginal coverage. Despite adaptive scaling could improve coverage for low-support transitions, it negatively affects the performance on high-support samples and increases the width of bounds. With 5-seed experiment done on HalfCheetah Medium, authors demonstrate that uncertainty magnitude, statistical calibration and downstream control utility can diverge, and marginal calibration cannot guarantee better policy performance.

Critical strengths
  1. Unlike most conformal‑for‑RL works which only reports marginal coverage as metric, this work focuses on how coverage varies between low‑support and high‑support state‑action pairs, as well as sharpness. Such analysis should be used more broadly in conformal‑for‑RL research community.
Critical weaknesses
  1. Experiment shows that the deterministic actor mean achieves a much higher return compared with unpenalized Q reranking. Consequently, the observed failure of calibrated penalties to improve return may be dominated by the candidate selection instead of the quality of the calibrated uncertainty estimates.
  2. The paper uses transition-level conformal quantiles while claiming that transitions within trajectories are temporally dependent. Avoiding episode leakage by splitting complete episodes across training, calibration, and validation sets sounds reasonable, but does not establish exchangeability among transitions within an episode.
Questions for authors
  1. Can the authors add more experiments that separate candidate generation quality from action ranking quality?
  2. How do the reported coverage rates and sharpness change under episode-level calibration? Would the main conclusions regarding support-conditioned reliability and marginal coverage still hold when the calibration procedure accounts for temporal dependence?
  3. I understand that the authors claim that they believe a case study on HalfCheetah alone is sufficient. But I am curious about the experiments on other tasks. What makes the authors stop from further expanding the scope?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.