- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies whether uncertainty-based pessimism in offline reinforcement learning is statistically reliable and whether a calibrated uncertainty can improve action selection. They propose a held-out one-step Bellman-residual proxy and use a split-conformal procedure. Using HalfCheetah Medium, they evaluate the calibrated bounds as penalities and show that they do not improve downstream control. The main conclusion is that uncertainty magnitude, statistical calibration, and downstream control utility should be evaluated separately.
Critical strengths
This is a meaningful and practically relevant question by separating uncertainty magnitude, statistical calibration, and downstream control utility. They are often implicitly conflated in offline RL.
Critical weaknesses
- The empirical study is restricted to a single HalfCheetah Medium task with a IQL backbone and five training seeds. It is unclear whether the conclusions generalize across environments or learning algorithms.
- Tha main phenomenon is not particular novel. The central observation that marginal conformal coverage cannot guarantee subgroup-conditioned miscoverage. This is already a well-known limitation of marginal conformal validity.
- The downstream control experiment is difficult to interpret. The unpenalized Q-reranking interface itself performs much worse than the original deterministic actor(from 12115 to 3640). How to interpret the failure in policy control as a limitation of calibrated pessimism itself.
- The control experiment fixes the penalty weight at , even though the calibrated constructions have substantially different bound widths.
Questions for authors
See the weaknesses