The paper studies how to select an online reinforcement-learning algorithm using an offline dataset and a fitted simulator. The Plug-In rule evaluates every candidate in one fitted simulator. The proposed Uncertainty-Aware (UA) rule instead constructs a bootstrap ensemble of simulators and selects the candidate with the largest average return across the ensemble.
The theoretical analysis considers the exploration parameter of UCB and Thompson Sampling in a two-armed Gaussian bandit. Theorem 1 shows that θ<σ2 gives regret T1−θ/σ2+o(1), whereas θ≥σ2 gives logarithmic regret with leading constant 2θ/(μ1−μ2). Proposition 2 shows that, under the paper's population parametric-bootstrap model, UA asymptotically selects at least as much exploration as Plug-In. Theorem 3 then shows no worse asymptotic regret growth and a polynomial-to-logarithmic improvement on an event whose probability tends to 1/4. Proposition 4 expresses ensemble averaging as a second-order curvature correction to the Plug-In objective.
The experiments include two-armed bandit simulations and PPO reward-weight selection for Hopper and Walker2d. UA has a higher mean true-environment score in all six deep-RL task and data-size settings, with significant paired differences in three.