acceptodds
Withdrawn from ICLR 2027

Uncertainty-Aware Selection of Online Algorithms with Simulator Ensembles

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The performance of online reinforcement learning depends critically on design choices, especially those that affect exploration. These choices are often selected by fitting a simulator to offline data, evaluating candidate algorithms in that simulator, and deploying the best-performing one. The simplest Plug-In selection rule simply selects the best performing algorithm on the fitted simulator, making evaluations unreliable when the offline data used to fit the simulator are limited. In this work, we investigate Uncertainty-Aware selection, which forms an ensemble of simulators—for example, obtained by bootstrap resampling—and selects the online algorithm with the best average performance across the ensemble. While ensemble-based approaches have been used to mitigate distribution shift and facilitate sim-to-real transfer, we formally show that this approach can mitigate the effects of limited data when fitting the simulator and theoretically has significant regret gains compared to Plug-In selection in multi-armed bandits. We also empirically investigate the Uncertainty-Aware selection approach in deep RL experiments on robotic control tasks that involves selecting reward-shaping hyperparameters, and show that it leads to more reliable selection and improved online performance.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Domain Randomization

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_LVS4OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary
  • The paper studies uncertainty-aware offline selection among candidate online learning algorithms. It constructs an ensemble of simulators reflecting uncertainty from finite offline data and selects the algorithm with the highest average performance across the simulator ensemble.
  • In a two-armed Gaussian bandit, it shows theoretically that accounting for simulator uncertainty can protect against under-exploration.
Critical strengths
  • Clarity of the paper: easy to follow and well-presented. The figures are extremely clear.
  • Theoretical contribution of using an ensemble in a two-armed Gaussian bandit situation.
Critical weaknesses
  • Misleading presentation: some presentation (section and paragraph titles) overgeneralise a phenomena proved only for a highly specific setting e.g. Proposition 2 establishes greater exploration only for the scalar exploration parameter of UCB/TS in a two-armed Gaussian bandit, but does not imply that UA systematically increases exploration. While the authors acknowledge this in Section 4, the paper would benefit from tightening its claims, which could otherwise mislead a reader.
  • Limited scope of theoretical results: the bandit result has value, but is restricted to two-arms, with Gaussian rewards, and UCB/TS. Proposition 4 is more general, but is a local characterisation of the ensemble objective rather than a guarantee that UA improves selection or performance in MDPs.
  • Limited empirical evidence: the empirical evaluation is narrow in task and selection diversity. Both deep-RL environments use the same MuJoCo simulator system-identification setup, the same algorithm (PPO) and design choice (reward-shaping). The experiments provide limited evidence for the paper’s broader framing of online algorithm selection (e.g. architecture and hyperparameters, as motivated in the introduction). The paper would benefit from more empirical evidence.
  • Limited algorithmic novelty: prior work has already proposed using an ensemble for hyperparameter tuning (e.g. Fully Offline Reinforcement Learning, Fellows et al., RLC 2026) (but look at a larger range of diagnostics e.g. median, variance of returns etc. rather than just the expected return). While they then train the algorithm offline, I am still concerned with the novelty of the idea.
Questions for authors
  • How would the theory extend to multiple arms (rather than only 2-armed bandits)?
Reviewer_9LxxOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies the problem of selecting an online RL algorithm by evaluating candidates in a simulator fitted to limited offline data. Offline data and deployment are assumed to come from the same environment, so finite-sample estimation is the source of error. The paper compares an uncertainty-aware (UA) estimator, based on an ensemble of simulators, with a single fitted simulator (Plug-In).

In a bandit setting, the paper shows that the regret is asymmetric in the exploration parameter (thm1), and that the UA estimator favors exploration and never has worse regret than the Plug-In estimator (thm3). The author(s) also provide a local curvature interpretation of the UA estimator's behavior, showing it can be viewed as a corrected Plug-In estimator (prop4). Deep RL experiments on MuJoCo environments show that UA estimators lead to higher success rates.

Critical strengths
  • Isolating estimation uncertainty from sim-to-real shift makes the problem clean, and the asymmetric-regret insight and its implications for selection are established theoretically.

  • The paper establishes regret bounds for UCB and Thompson sampling that hold uniformly over the exploration parameter θ\theta and the suboptimality gap (lemmas 8 and 14), going beyond the fixed-parameter characterization (thm1), which is closely related to Fan & Glynn (2025). These uniform bounds are needed for the selection analysis (prop2 and thm3).

Critical weaknesses
  • The contribution appears too narrow. Scoring candidates by their average over an ensemble of plausible simulators, rather than on a single fitted one, is a standard way of dealing with model uncertainty. The same idea is used in robust RL and domain randomization. The contribution is the regret analysis, which is closely related to Fan & Glynn (2025).

  • UA is compared only to Plug-In. Simple conservative alternatives are missing, such as Plug-In with an upper-confidence variance estimate, a one-step shift toward more exploration, or posterior sampling of simulator parameters.

  • The deep RL experiment doesn’t test the paper’s theory. It measures the final policy’s score, not regret during learning. The exploration trade-off behind the bandit results never comes into play, and the simulator is a lookup table with interpolation, which breaks the smoothness prop4 assumes. The experiment shows a small practical gain, but it doesn’t support either theoretical claim.

Questions for authors
  • The bandit results make one clear point: when under-exploring costs far more than over-exploring, Plug-In lands on the costly side, and UA avoids it. The rest of the paper does not build on this point. Proposition 4 is a general curvature result unrelated to the asymmetry, and the deep RL experiment tunes reward weights for final performance, where no such asymmetry is present. Aligning the general result and the experiments with the bandit claim would make the paper clearer and more convincing.

  • If the cost of mis-tuning is the same in both directions, does UA still outperform Plug-In? Testing this would show whether the gain comes from the asymmetry or from something else.

Reviewer_9hsyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether averaging over bootstrap simulators improves the choice of an online learner when offline data are limited. In two-armed Gaussian bandits, it shows that underexploration can lead to polynomial regret and that population Uncertainty-Aware (UA) selection can retain logarithmic regret in cases where Plug-In selection incurs polynomial regret. A local curvature expansion describes how ensemble averaging changes candidate scores. Experiments report gains in bandit exploration tuning and PPO reward shaping.

Critical strengths
  • S1. A concrete failure mode for Plug-In selection. The bandit analysis explains why errors in exploration tuning have asymmetric costs. Too little exploration can change logarithmic regret into polynomial regret, while extra exploration above the threshold changes the leading logarithmic constant.

  • S2. A matched-budget comparison beyond bandits. Both PPO selectors receive 12 training runs per candidate and use the same offline data within each replication. UA improves mean success in all six settings, with three differences marked significant.

Critical weaknesses
  • W1. The regret-rate guarantee leaves the cost of extra exploration unresolved. By Theorem 1, always choosing the upper endpoint θˉ>σ2\bar\theta>\sigma^2 also yields logarithmic regret and achieves the same order improvement whenever Plug-In has polynomial regret. Eq. (9) tends to one when both regrets grow logarithmically, even if their leading constants differ substantially.

  • W2. The bandit experiments test a different bootstrap from the one analyzed. Eqs. (7)–(8) average over mean perturbations with arm-specific variances held fixed. If the higher-variance arm has the lower fitted mean, a finite ensemble may miss the rare draws in which that arm becomes optimal. Figure 2 instead refits a pooled variance along with the means, leaving open whether the mechanism in Theorem 3 operates at practical ensemble sizes.

  • W3. The PPO comparison leaves the role of dataset-specific uncertainty unclear. Table 1 compares a single fitted simulator with a bootstrap ensemble. The gains could come from averaging over perturbed simulators even if the perturbation covariance were shared across offline datasets.

Questions for authors
  • Q1(W1). Can you show a finite-horizon regret or leading-constant advantage for UA over always choosing θˉ\bar\theta?

  • Q2(W2). At the horizon and offline sample size of Figure 2, how does regret vary with ensemble size for the mean-only bootstrap in Eq. (7) and for the pooled-variance bootstrap used in the experiment?

  • Q3(W3). On Walker2d, at the same 12-run budget, how does UA compare with an ensemble centered on the same fitted simulator whose perturbation covariance is fixed across datasets and matched to the average bootstrap covariance at each offline sample size? Please report the paired difference and its confidence interval.

Required Corrections

  • p. 2, ll. 102–107. Define the history as a chronological sequence and use a distinct within-episode index for past episodes.

  • p. 4, ll. 211–215 and p. 5, ll. 238–243. State how each arm is initialized. At zero pulls the UCB index is undefined and the flat-prior TS posterior is improper.

  • p. 6, ll. 313–315 and p. 7, l. 367. Reconcile the allocation of ⌊Toff/2⌋\lfloor T_{\mathrm{off}}/2\rfloor observations per arm with Theorem 3(i), which also covers odd ToffT_{\mathrm{off}}.

  • p. 8, ll. 412–417. Describe Eq. (11) as a correction to each candidate’s score. A candidate’s score can decrease while its relative rank improves.

  • p. 6, l. 274 and p. 7, ll. 337–339. Use “Lemmas 8 and 14,” “the contour lines … depict,” and “rightmost.”

  • p. 9, l. 471. Fix the C.8 link, which points to the Table 6 anchor in the reviewed PDF.

Reviewer_x23SOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how to select an online reinforcement-learning algorithm using an offline dataset and a fitted simulator. The Plug-In rule evaluates every candidate in one fitted simulator. The proposed Uncertainty-Aware (UA) rule instead constructs a bootstrap ensemble of simulators and selects the candidate with the largest average return across the ensemble.

The theoretical analysis considers the exploration parameter of UCB and Thompson Sampling in a two-armed Gaussian bandit. Theorem 1 shows that θ<σ2\theta<\sigma^2 gives regret T1−θ/σ2+o(1)T^{1-\theta/\sigma^2+o(1)}, whereas θ≥σ2\theta\geq\sigma^2 gives logarithmic regret with leading constant 2θ/(μ1−μ2)2\theta/(\mu_1-\mu_2). Proposition 2 shows that, under the paper's population parametric-bootstrap model, UA asymptotically selects at least as much exploration as Plug-In. Theorem 3 then shows no worse asymptotic regret growth and a polynomial-to-logarithmic improvement on an event whose probability tends to 1/41/4. Proposition 4 expresses ensemble averaging as a second-order curvature correction to the Plug-In objective.

The experiments include two-armed bandit simulations and PPO reward-weight selection for Hopper and Walker2d. UA has a higher mean true-environment score in all six deep-RL task and data-size settings, with significant paired differences in three.

Critical strengths
  1. Theorem 1 gives a clear explanation for why simulator error can be costly when selecting an exploration parameter. For both UCB and Thompson Sampling, underexploration changes the order of regret from logarithmic to polynomial, while overexploration changes only the leading logarithmic constant. Lemmas 8 and 14 establish uniform bounds that allow the selected parameter to depend on the offline data and the deployment horizon.

  2. The UA rule is simple and treats candidate learners as black boxes. Plug-In and UA receive the same simulation budget and are evaluated on the same offline datasets, so the paired differences isolate the selection rule. The paper reports lower-tail outcomes as well as means. The additional bandit experiments show the exploration asymmetry for 16 combinations of reward gap and noise, and the deep-RL experiment shows the same qualitative advantage for two tasks and three offline-data sizes.

Critical weaknesses
  1. The theoretical improvement does not analyze the procedure used in the bandit experiments. Theorem 3 fits a separate variance for each arm and perturbs only the fitted means. If a^\hat a is the fitted optimal arm, Plug-In converges to the projection of σ^a^2\hat{\sigma}_{\hat a}^2, while UA is lower-bounded by the projection of max⁡(σ^12,σ^22)\max(\hat{\sigma}_1^2,\hat{\sigma}_2^2). The strict-improvement event is σ^a^2<σ2<σ^3−a^2\hat{\sigma}_{\hat a}^2<\sigma^2<\hat{\sigma}_{3-\hat a}^2. Appendix B.3 instead uses one pooled variance, random arm counts, and a finite bootstrap that re-estimates the pooled variance in every member. Therefore, Figures 2, 3, and 6 do not test the mechanism proved in Theorems 7 and 13. There is also no guarantee for finite ensemble size. The polynomial-to-logarithmic result first sends TT to infinity for each fixed ToffT_{\mathrm{off}} and only then sends ToffT_{\mathrm{off}} to infinity, so it does not show whether the separation persists under a practical joint scaling of offline data and deployment horizon.

  2. The paper interprets UA as favoring candidates that perform more consistently across plausible simulators, but its mean-aggregation rule does not directly reward consistency. Proposition 4 only shows that ensemble averaging adds a curvature-dependent correction, which can have either sign and need not improve performance in the true environment. Moreover, the regret guarantee is specific to two-armed Gaussian bandits and does not establish a similar advantage for general MDPs. The empirical results are encouraging, but they do not by themselves support the paper's broader reliability claims.

Questions for authors
  1. Does Proposition 2 or Theorem 3 continue to hold for the procedure in Appendix B.3, where the simulator has a pooled variance, arm counts are random, and each bootstrap member re-estimates the pooled variance? If so, what replaces the max-of-arm-variances argument used in Theorems 7 and 13?

  2. What guarantee is available for a finite ensemble, and what happens when TT and ToffT_{\mathrm{off}} grow jointly? The current 1/41/4 statement takes the deployment-horizon limit before the offline-sample-size limit.

  3. What is meant by ‘consistent’ here? Mean aggregation does not penalize variation across simulators, and Proposition 4's correction can have either sign. Under what additional conditions does that correction improve true-environment selection?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.