acceptodds
Under review as a conference paper at ICLR 2027

Optimistic Preference Exploration with Finite Reward Ensembles

Abstract

Learning systems increasingly choose the feedback and experience used to improve them. In preference learning, a comparison can favor a strong current response or reveal information that helps future decisions. We develop Optimistic Learning with Ensembles (OLE), a framework connecting finite reward sampling, comparison selection and feedback updates. Its central idea is to assess reward ensembles through the learning decisions they induce. Our analysis establishes a query-regret guarantee for optimistic posterior sampling in a bounded linear preference model. An exact construction explains when information acquired by exploration repays its immediate cost and how dependence between reward samples changes this tradeoff. For implementation, we identify the relative score errors that govern the precision of a chosen selection rule. Controlled learning studies examine the interaction between sampling and selection, while neural experiments demonstrate more precise implementation at a fixed reward-evaluation budget. Agentic OLE applies reward-sensitive selection to generated training trajectories, with evidence from question answering, research and interactive tasks. Together, the results connect preference exploration to the broader design of learning systems that choose their own experience.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Preference learning, Posterior sampling, Reward ensembles, Active learning, Agentic learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rQq3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors propose Optimistic Learning with Ensembles (OLE). It focuses on the intersection of optimistic sampling, active preference learning, and reward ensembles. Specifically, it uses finite reward ensembles to select which preference comparisons or training experiences should be acquired. The experiments show improvement on the neural reward model and agentic setting.

Critical strengths
  • The authors identify an underexplored distinction in the intersection between reward ensemble and data selection.

  • The motivation is theoretically grounded, and the main theoretical treatment is careful. The two-round construction makes the exploration tradeoff concrete.

Critical weaknesses
  • The paper currently feels like several related papers placed under one umbrella, and the connection between the theoretical OLE object and the later experiments is weak. Sections 2–3 use posterior reward-function draws to evaluate the same candidates, but Section 5.2 uses no learning or posterior update. Moreover, agentic OLE only uses variation comes from realized rewards across generated trajectories with no posterior preference-learning loop present either.

  • The strongest positive preference-learning result appears highly dependent on a deliberately selected “informative challenger” geometry. With OpenAssistant geometry, the optimism seems not helpful, even when the labels are replaced with correctly specified synthetic BTL feedback, mean selection has lower regret. Thus, OLE seems not generally preferable for preference acquisition.

  • The Agentic OLE experiments contain major attribution and reproducibility problems. The appendix states that it is unresolved whether the rejection ablation used trajectory-level selection or prompt-group selection, so the experiment cannot identify which selection rule caused the gain. Given that this is the portion of the paper that most directly supports the relevance to modern LLM agents, I think these issues need to be resolved rather than only documented as limitations.

  • The writing flow needs substantial improvement:

    • The paper should give an intuitive understanding of this optimistic reward ensemble. Even after finishing reading the introduction and the beginning of Section 2, it is not clearly presented why this optimism is important in reward ensemble and pair selection.

    • A formulation of the task focus is preferred before Section 2. Currently, the method directly begins without clearly presenting the problem definition.

Questions for authors
  • What exactly is the common mathematical object linking Agentic OLE to Sections 2–4? Can the trajectory-selection rule be written as an instance of the same finite-ensemble selection objective, or should Agentic OLE instead be presented as a heuristic inspired by OLE?

  • Which rejection implementation produced the reported Agentic OLE numbers? The appendix currently says this is unresolved. This seems essential to resolve before publication. A matched-volume mean-only rejection baseline in the agentic experiments would be helpful. This would directly test whether the dispersion/optimism component contributes beyond filtering low-reward trajectories.

  • Can the paper provide a practical statistic, computed before collecting labels, that predicts whether a candidate set resembles the “informative challenger” regime in which optimism is expected to repay its cost? This would turn the negative natural-geometry results into a stronger scientific conclusion.

  • Is there a realistic sequential learning experiment in which balanced sampling's reduction in selector approximation error produces a measurable downstream learning gain? This would substantially strengthen the bridge between Section 4 and the broader OLE motivation.

9 comments
AuthorsOpenReview ↗

Rebuttal for W1

The common question is how to select data for learning under a finite reward-evaluation budget, and how accurately that selection can be implemented. The components examine the learning consequences of selection, the implementation precision of finite ensembles, and an application to selecting policy-training experience. They share the scoring form

Fa(v)=mean⁡(v)+a sd⁡(v),F_a(v)=\operatorname{mean}(v)+a\,\operatorname{sd}(v),

with the source of vv and the selected object specified separately:

ComponentSource of reward vectorSelected objectUpdate or evaluation target
Preference learning, Sections 2–3Reward functions' predictions of the same candidate's gap relative to the referenceNext preference comparisonLabels update the posterior; query regret is analyzed
Implementation precision, Section 4 and original Section 5.2Finite evaluations under a fixed reward lawApproximation to a specified soft selectorRelative score errors and selector KL
Additional neural closed-loop studiesPredictions from the current reward headNext preference-query batchLabels update the head; predictive quality is evaluated
Additional agentic controlled studiesRealized rewards of completed trajectories for one promptComplete prompt groupsRetained experience updates the policy

Why the fixed-model neural study connects to the theory. Section 4 fixes the reward law and target selection rule, then analyzes selection error from finite reward evaluations. Section 5.2 directly tests this question: at the same budget of 20 reward evaluations, does changing the sampling arrangement implement the target rule more accurately? Holding the model fixed isolates this sampling error. Its theoretical counterpart is the selector-precision analysis in Section 4. The additional closed-loop studies then examine learning outcomes when selection feeds into actual model updates.

Additional experiments complete the selection–feedback–update loop. The new neural studies select queries using the current reward head, obtain their preference labels, update the head, and use it for subsequent selection. In particular, the ActiveUltraFeedback experiments update after each batch of 32 labels, with each sampled mask shared across candidates of the same prompt.

On helpful-online, using recorded human preferences, each method receives 896 labels across 128 paired seeds. Holding the sampler fixed, optimistic scoring reduces final NLL relative to mean scoring:

SamplerFinal NLL reduction: mean minus optimisticAdjusted seed-level 95% CI
IID0.008437[0.006737, 0.010138]
Balanced0.010201[0.008415, 0.011986]

These exploratory contrasts were evaluated after the primary analysis and corrected as a separate four-comparison family. They directly evaluate how the selection rule affects predictive quality through actual reward-head updates at a matched label budget.

Connection to the agentic application. In the additional group-filtering experiments, trajectory rewards are available before retention, and the mean-plus-dispersion score determines which prompt groups enter policy updates. The updated policy generates the next round of trajectories, linking experience selection to policy learning. Here, dispersion describes realized reward variation across trajectories. Theorem 3.1 provides the guarantee for posterior sampling under the bounded linear-logistic model; matched policy-training controls evaluate the learning effects of the agentic application.

AuthorsOpenReview ↗

Rebuttal for W2

The additional experiments extend the evidence to OpenAssistant and test whether favorable conditions for optimism can be identified before new feedback. This complements the exact two-round analysis in Section 3.2: exploration reduces cumulative regret when the future decision benefit of additional information exceeds its immediate query cost.

1. Predictive gains on OpenAssistant human preferences. The new study uses frozen Qwen3-1.7B features and two-dimensional PCA fitted only on training data. Four methods run with 24 new paired seeds and 512 queries per learning trajectory. Each method acquires labels for its own selected comparisons and updates its reward learner. The prespecified primary endpoint is normalized NLL area under the learning curve:

SamplerMeanMax2Max2 minus mean, 95% CIHolm p
IID0.6841950.683535−0.000660 [−0.000840, −0.000480]3.15×10−73.15\times10^{-7}
Coupled0.6842380.683525−0.000713 [−0.000873, −0.000553]1.39×10−81.39\times10^{-8}

These 24 new seeds are analyzed separately from the earlier eight. Both samplers support a small predictive benefit from optimistic scoring, providing direct evidence beyond the informative-challenger construction. The endpoint here is predictive quality.

2. Sampling and exploration strength on OpenAssistant geometry. In a separate study with 128 BTL episodes and 512 queries per trajectory, holding the score fixed and replacing IID with coupled sampling reduces mean-rule regret from 10.0612 to 8.1639 (18.86%) and max2 regret from 10.2372 to 9.3991 (8.19%). These comparisons directly test sampling dependence. Within this study, coupled mean retains lower regret than coupled max2: 8.1639 versus 9.3991.

Independent confirmation also supports adjusting exploration strength. After freezing the development-selected dispersion coefficient of 0.25, evaluation on 128 new BTL episodes reduces coupled-rule regret from 9.0738 under the original coefficient of 1 to 8.0621, an 11.15% reduction. The paired difference is −1.0116, with 95% CI [−1.5193, −0.5040] and Holm p = 0.000793 across seven prespecified comparisons. This identifies the benefit of adapting exploration strength to the task conditions.

3. Identifying favorable conditions before new feedback. We freeze a posterior-forecast diagnostic and threshold on 48 development environments, then evaluate them on 160 independent simulated-BTL confirmation environments, equally divided between natural-text and informative-challenger geometries. This separate study uses posterior-moment scores with softmax selection at temperature 0.05.

The diagnostic achieves AUROC 0.7319, with 95% CI [0.6324, 0.8242]. Before confirmation feedback, the frozen threshold identifies 98 environments as favorable. Within these 98 environments, optimistic selection reduces cumulative regret relative to mean selection by 6.1335 on average, with multiplicity-adjusted 95% CI [4.5477, 7.7192]. This evaluates the benefit within a prospectively identified favorable subset. Our response to Q3 provides the full diagnostic protocol and results.

AuthorsOpenReview ↗

Rebuttal for W3

We have added controlled experiments with an explicitly specified filter, matched retained volume, and paired training runs. These experiments will provide the primary evidence for dispersion-specific claims in the revision. The available historical records do not yet uniquely identify which implementation produced the original rejection-ablation checkpoints; the new experiments execute the following identified operation.

Explicit group-level filtering. Each prompt produces four trajectories with rewards rb,1,…,rb,4r_{b,1},\ldots,r_{b,4}. We compute

\qquad s_b=\sqrt{\frac14\sum_{i=1}^{4}(r_{b,i}-\mu_b)^2}.$$ Mean+SD ranks prompt groups by $\mu_b+s_b$, mean-only ranks them by $\mu_b$, and random retention uses a random ranking. All four trajectories in each selected group are retained. Within each study, filtered methods match initialization conditions, action interfaces, generated-trajectory counts, retained-trajectory counts, and update counts. The mean+SD versus mean-only comparison therefore directly tests the contribution of adding dispersion to the ranking score. **Results from 512 steps and eight paired seeds.** Each step generates 16 groups and retains eight or four, corresponding to 50% or 25% retention. Each arm and seed generates 32,768 trajectories and retains 16,384 or 8,192, respectively. Against mean-only at the same retained volume: | Retention / evaluation | Mean+SD minus mean F1 (pp) | Adjusted paired-seed 95% CI | |---|---:|---| | 50% / primary evaluation | +3.21 | [0.60, 5.82] | | 25% / primary evaluation | +7.03 | [5.47, 8.59] | | 50% / 2,048 additional questions | +3.29 | [2.13, 4.45] | | 25% / 2,048 additional questions | +8.29 | [6.71, 9.87] | The primary and additional-question evaluations are adjusted within separate four-comparison families. The additional questions evaluate the same trained models. These comparisons support a final-answer F1 benefit from the dispersion term over mean-only filtering in this setting. **Other training settings.** In the 100-update study with six paired seeds, the mean+SD-minus-mean-only F1 difference is −0.0368 pp, with 95% CI [−0.7201, 0.6250]. In the 50-update study with three matched warm starts, the corresponding difference is −0.2423 pp, with 95% CI [−0.9960, 0.4864]. The latter study also improves answer validity over random retention by 0.9766 pp, with 95% CI [0.0326, 2.0508], identifying an answer-format benefit. These studies have different reward, normalization, and initialization settings and will be reported separately. **Reproducibility documentation.** For the new experiments, the revised appendix will provide filtering pseudocode and associate each run configuration, seed, checkpoint, and per-seed result. It will also specify reward and advantage definitions, generation and retention counts, and evaluation protocols, so that each filtering claim is tied to an identified executed procedure.
Reviewer_P16uOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper studies optimistic learning where reward ensembles are sampled from the posterior distribution which is updated at every step based on the reward obtained from the sampled prompts and responses. The paper calls it optimism as the reward ensemble that gives the best reward guides the sampling of the prompts. The authors provide Baysian regret guarantees on this framework and perform experiments under the agentic framework ranging from question answering to research.

Critical strengths

The theoretical guarantee and setup is easy to follow and seems reasonable though I did not carefully verify the details in the appendices. The experiments in table 1 A demonstrate that the regret that the proposed algorithm attains is comparable to the popular Thompson sampling algorithm as well.

Critical weaknesses

I divide the weakness of the paper into two parts a) Poor presentation and b) Lack of comparisons with proper literature

Poor presentation:

The presentation of the the paper leaves much to be desired. A lot of technical terms have been used in the paper without defining them or pointing the reader to the right appendix section. Let me give some examples below.

a) The terms Section 3.2 starts using terms like "safe" gap and "informative" gap. These terms have not been defined in the paper at all yet. Again the next line says second round "offers" gaps z/2 and -z/2. While one might be able to guess the second line that it denotes the gap from the base prompt y0y_0, using the term of offering gap is weird as gap is not directly observed.

b) In the theoretical section, it should have been mentioned that ψ(xt,y)\psi(x_t,y) is known to the algorithm/learner which would be used in computing the posterior of θ∗\theta^\ast after observing the samples. But this is not clearly mentioned.

c) The notion of "coupling" is not defined clearly at all which is extensively used in the experiments and some intermediate sections too. While coupling is introduced in section 2, which characterizes the joint distribution of the the reward ensembles {θ1:N}\{\theta^{1:N}\}, it is indirectly used in section 3.2 too as well but never clearly described. I think "Pairing opposite posterior quantiles preserves each reward function’s sampling law and improves the use of acquired information in this example" on line 235 kind of talks about a joint distribution but it is never clearly defined as a coupling at all. This problem is continued in section 4 as well where the notion of rejection pairs is not defined at all.

The issue gets deeper in the experimental section I did not find any clean definition of how it is coupled as well.

d) I also could not find a clear setup for the "memory" action as well. It is used in figure 2 as a label but I could not find a clean description for it.

Missing literature

While the authors claim that they build on the Thompson sampling literature to get bounds for the optimistic instance where more than a single reward model is sampled, I would like to point out that this framework is not new and has been studies in the linear contextual bandit literature. For example, see the cited papers [1,2].

Optimistic sampling has already been proposed in this paper where multiple θ(i)\theta^{(i)} are sampled at every step under a more sophisticated multinomial reward model. If I am not mistaken, the logistic model considered in this paper should reduce to a special case of their paper when we just restrict the number of arms sampled at every step to unity.

[1] Agrawal, S., Avadhanula, V., Goyal, V., & Zeevi, A. (2017). Thompson sampling for the MNL-bandit. Proceedings of the 2017 Conference on Learning Theory, 65, 76–78. [2] Oh, M.-h., & Iyengar, G. (2019). Thompson sampling for multinomial logit contextual bandits. Advances in Neural Information Processing Systems, 32, 3145–3155.

Questions for authors

I have no specific questions at this stage, but I would appreciate it if the authors could clarify any points in my review that I may have misunderstood or overlooked.

8 comments
AuthorsOpenReview ↗

Rebuttal for W1–Part I

Thank you for requesting operational definitions of the safe and informative responses and clarification of what the learner observes in round two. The construction specifies candidates through known comparison features; their reward gaps depend on those features and the unknown reward parameter.

Specifically, the unknown parameter is θ⋆=(c,z)\theta^\star=(c,z), with a known uniform prior on [1/2,3/2]×[−1,1][1/2,3/2]\times[-1,1], and 0<δ<1/80<\delta<1/8. The reference is a response with reward 1/21/2. Each candidate’s reward gap relative to the reference satisfies

gθ⋆(y)=ψ(y)⊤θ⋆.g_{\theta^\star}(y)=\psi(y)^\top\theta^\star.

Round / responseKnown comparison feature ψ(y)\psi(y)Latent reward gap
First-round safe response(δ,0)(\delta,0)δc\delta c
First-round informative response(0,1/2)(0,1/2)z/2z/2
Second-round response 1(0,1/2)(0,1/2)z/2z/2
Second-round response 2(0,−1/2)(0,-1/2)−z/2-z/2

Meaning of the names. The safe response has a positive gap relative to the reference throughout the prior support. Its prior-mean gap is δ\delta, compared with zero for the informative response. Its first-round preference label provides information about cc; under this product prior, that label leaves the posterior over zz unchanged.

The informative response provides a label about zz. Since the ranking of the second-round candidates depends on the sign of zz, this information can guide the subsequent choice. The names therefore describe the candidates’ immediate-reward and future-information roles.

What is observed. The learner evaluates the known features under sampled reward parameters to select a comparison, then receives a binary preference label. The numerical values of the true reward gaps in the table are not supplied as observations.

We will replace the second-round description with:

In round two, the learner receives two candidate responses with known comparison features (0,1/2)(0,1/2) and (0,−1/2)(0,-1/2). Under the unknown parameter θ⋆=(c,z)\theta^\star=(c,z), their latent reward gaps relative to the reference are z/2z/2 and −z/2-z/2, respectively.

These definitions will appear at the start of the construction in Section 3.2, making the relationship between candidates, known features, latent rewards, and observed feedback explicit.

AuthorsOpenReview ↗

Rebuttal for W1–Part II

Thank you for asking us to make the feature map’s availability explicit. In the theoretical model, the learner knows the prior and the comparison features ψ(xt,y)\psi(x_t,y) for every candidate; the unknown quantity is the reward parameter θ⋆\theta^\star.

At each round, the prompt, candidate set, and reference yt0y_t^0 are determined before current reward sampling. They may depend on the observed history but carry no additional information about θ⋆\theta^\star conditional on that history.

Known features support scoring and likelihood evaluation. For any parameter θ\theta, the candidate’s reward gap relative to the reference is

=r_\theta(x_t,y)-r_\theta(x_t,y_t^0) =\psi(x_t,y)^\top\theta.$$ The learner samples reward parameters using its current posterior inference and selects a candidate $y_t$. It then receives a binary preference label $b_t$, where $b_t=1$ indicates preference for the candidate over the reference, with $$\Pr(b_t=1\mid\theta^\star,G_t,y_t) =\sigma\!\left(\psi(x_t,y_t)^\top\theta^\star\right), \qquad \sigma(u)=\frac{1}{1+e^{-u}}.$$ **The observed label updates the posterior.** Writing $\psi_t=\psi(x_t,y_t)$, the Bayesian update is $$\Pi_t(d\theta)\propto \Pi_{t-1}(d\theta)\, \sigma(\psi_t^\top\theta)^{b_t} \left[1-\sigma(\psi_t^\top\theta)\right]^{1-b_t}.$$ The known $\psi_t$ makes the label likelihood evaluable for any hypothesized parameter $\theta$, while the observed $b_t$ supplies the new evidence for updating. Subsequent reward sampling targets the updated posterior $\Pi_t$, either exactly or approximately. We will state the known-feature condition explicitly in Section 2.1 and make the order of sampling, comparison selection, label observation, and posterior updating explicit in Algorithm 1.
AuthorsOpenReview ↗

Rebuttal for W1–Part III

Thank you for requesting a consistent definition of coupling and concrete descriptions of its implementations. A coupling is a joint distribution with prescribed marginals; it specifies how random draws depend on one another. In posterior preference acquisition, conditional on the observed history and current inputs,

\qquad \theta_t^{(1)}\sim\nu_t,\quad \theta_t^{(2)}\sim\nu_t.$$ Here, $\nu_t$ is the prescribed sampling law, either the exact posterior or its approximation, and $\kappa_t$ determines dependence between the draws. IID sampling corresponds to $\kappa_t=\nu_t\otimes\nu_t$. Each parameter draw defines a reward function evaluated on the same candidates and reference. **Quantile pairing in Section 3.2.** In the two-round construction, the current posterior factorizes over $c$ and $z$. Let $F_c,F_z$ denote the posterior-factor CDFs. Each round draws fresh independent $U_c,U_z\sim\mathrm{Unif}(0,1)$ and constructs $$\begin{aligned} \theta^{(1)} &=\bigl(F_c^{-1}(U_c),F_z^{-1}(U_z)\bigr),\\ \theta^{(2)} &=\bigl(F_c^{-1}(1-U_c),F_z^{-1}(1-U_z)\bigr). \end{aligned}$$ Both $U$ and $1-U$ are uniform, so each member preserves the correct posterior marginals. Independent randomness across coordinates gives each member the full product-posterior law. This is the specific coupling described as pairing opposite posterior quantiles. **Paired rejection sampling in Sections 4.1 and 5.1.** Two rejection-sampling streams use paired symmetric proposal innovations $Z,-Z$ and complementary acceptance uniforms $U,1-U$, independent of the proposal randomness. Each stream applies its usual acceptance test and stops at its own first acceptance. Once one stream accepts, its output is retained while the other continues. Viewed individually, each stream still has the correct proposal and uniform distributions and performs ordinary rejection sampling. Consequently, each output has the target posterior marginal in exact arithmetic, including for asymmetric posteriors. The shared randomness determines dependence between the outputs. **Balanced dropout in Section 5.2.** Within a block of $b$ members, each mask coordinate $\ell$ independently selects one uniformly random member $J_\ell$ in which to drop that coordinate: $$J_\ell\sim\mathrm{Unif}\{1,\ldots,b\}, \qquad D_{i\ell}=\mathbf{1}\{J_\ell=i\}.$$ Each member therefore retains independent dropout coordinates with drop probability $1/b$, while masks are dependent across members. The original neural experiment organizes 20 evaluations into two ten-member blocks, preserving dropout probability 0.1 for each member. **Sharing across candidates.** Posterior preference acquisition evaluates all candidates using the same parameter draw. The original RewardBench 2 precision study uses candidate-specific dropout streams; the additional nonlinear-head studies share each mask across all candidates of a prompt. We will explicitly identify this setting in each experiment. The revision will define coupling at first use and specify each construction’s joint randomness, preserved marginals, and sharing across candidates in the corresponding sections.
Reviewer_pUaBOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes optimistic learning with ensembles (OLE), a framework for selecting responses to collect preference labels in order to improve reward learning. The main idea is to use optimism under uncertainty, similar to UCB, to favor responses that could provide useful information for future learning. This is done by estimating both the expected reward and uncertainty of different responses using an ensemble of reward functions. The paper provides regret bounds for this optimistic selection rule using two reward samples and also studies the approximation error of the selection rule when using sampled reward estimates. The experiments include a controlled task using frozen Qwen2.5 features and synthetic preference labels, a neural network based reward model using dropout to generate different reward estimates, and agentic LLM tasks using reward-based filtering of trajectories before policy updates.

Critical strengths

None

Critical weaknesses

Limited contributions. The main idea of using optimism under uncertainty is already well established in methods such as UCB, and using ensembles to estimate uncertainty is also not new. The proposed selection rule (Eq. 2) is essentially a UCB-style score computed using reward samples. In the two-sample case, it reduces to selecting the maximum of two sampled rewards (Eq. 6), which the paper itself acknowledges is not new. The paper provides regret bounds for this selection rule, but I am not sure what additional advantage it establishes over existing methods such as UCB or Thompson sampling. In particular, I am not sure how the bound directly establishes that optimism or coupling improves regret over these existing methods.

Presentation and Experiments. I find the paper very difficult to follow, particularly in understanding the proposed method and how the different theoretical and experimental components fit together. The main text frequently introduces terminology without sufficient explanation. Some examples (among the many) are terms such as "mean-plus-dispersion selector," "matched projection work," and "retained-ensemble study" that appear before they are properly explained. Similarly, the experiments use terms such as "informative-challenger rule" and "encoder/orientation cell" without clearly describing what they mean. The theoretical results are also difficult to interpret, with several constants and assumptions introduced without sufficient intuition. I think the main paper should be understandable without requiring the reader to go through the appendix to understand the method and experiments.

Another major concern is the connection between the theoretical analysis and the experimental implementation. The theory considers posterior sampling of reward functions and updating the posterior using preference feedback. However, the neural network-based reward model experiments use dropout samples from a fixed reward model without posterior updates and only evaluate the approximation error of a selection rule. The agentic experiments are even further removed, using realized trajectory rewards to filter training experience before GRPO updates. It is unclear how the theoretical framework relates to these implementations.

I also have several concerns about the agentic experiments (Table 1C). It is unclear which rejection/filtering procedure is used in the reported ablations, and whether the baselines are compared under matched training, inference, and sampling budgets. More importantly, the experiments do not compare optimistic selection against simpler alternatives such as mean-only ranking or random filtering, making it difficult to isolate the contribution of optimism. I am also not sure how the proposed method differs from simply using a UCB-style selection rule, which has not been compared here. The results also lack per-seed uncertainty estimates. Given that these experiments are used to demonstrate the practical benefits of the proposed framework, I do not find the current comparisons sufficient to support those claims.

Questions for authors
  1. What advantage does OLE offer over Thompson sampling or UCB-based methods?

  2. How are the agentic experiments budget-matched against the baselines, and what rejection/filtering procedure is used?

8 comments
AuthorsOpenReview ↗

Rebuttal for W1

Thank you for asking us to clarify OLE’s contribution beyond existing optimism and reward ensembles. OLE investigates how the joint distribution of a finite reward ensemble affects preference acquisition and subsequent learning. At a given posterior, reward samples can have identical marginal distributions but different dependence, leading to different selected queries and feedback histories. Our theoretical and empirical results characterize this design choice.

A regret guarantee allowing dependence and posterior approximation. Under the bounded linear-logistic preference model, Theorem 3.1 provides a Bayesian query-regret guarantee for two fresh reward-function draws per round, allowing arbitrary dependence within the pair and explicitly accounting for marginal posterior approximation through a Wasserstein error term. This permits the design of sample dependence within the stated guarantee. Section 4 further derives the leading asymptotic approximation error of a fixed soft selection rule in terms of errors in candidate-score differences, providing a criterion for sampling design.

Direct comparisons establish when exploration and coupling reduce regret. Section 3.2 gives exact regret comparisons in a two-round preference-learning problem. Propositions B.3–B.4 quantify the immediate cost and subsequent benefit of an informative query, establishing conditions under which optimism improves over matched mean selection. More directly, Corollary B.5 proves that quantile-antithetic optimistic sampling has strictly lower expected cumulative regret than independent optimistic sampling throughout 0<δ<1/80<\delta<1/8 in this construction, while preserving each draw’s posterior marginal. The gain involves both query acquisition and the subsequent use of feedback. Thus, the paper provides both a guarantee across sampling designs and direct comparative results explaining their learning benefits.

Additional experiments isolate coupling with the score held fixed. We use OpenAssistant responses represented by frozen Qwen3-1.7B features and a two-dimensional PCA representation fitted only on training data. Across 128 simulated Bradley–Terry teacher episodes, each method uses two posterior function draws per round and 512 queries, following its own feedback-and-update history.

Fixed scoring ruleIID regretCoupled regretCoupled − IID, 95% CI
Two-sample mean10.06128.1639−1.8973 [−2.9256, −0.8689]
Original max2 optimism10.23729.3991−0.8381 [−1.3507, −0.3256]

The corresponding regret reductions are 18.86% and 8.19%, with Holm-adjusted p-values of 0.001522 and 0.004642. Each row holds the scoring rule, function-draw count, and label budget fixed, directly testing the effect of the joint sampling structure in this grid-posterior learner.

Direct comparisons with TS and UCB. The original informative-challenger experiment reports the following results on the same learning tasks:

MethodQuery regret ↓Final-decision utility shortfall (10−210^{-2}) ↓
Coupled mean19.821.79
Exact TS20.211.32
Local Fisher-UCB14.301.09
Coupled optimistic OLE17.431.02

OLE reduces cumulative query regret by 2.78 relative to exact TS and by 12.1% relative to matched coupled mean selection, and achieves the lowest final-decision utility-shortfall point estimate among these methods. These metrics respectively evaluate the quality of queried responses and the final deployment decision. We provide further baseline comparisons in our response to Q1.

We will foreground this sampling-design question and its corresponding guarantees, comparative results, and empirical evidence in the introduction and related work.

AuthorsOpenReview ↗

Rebuttal for W2

Thank you for requesting a self-contained explanation of the method and experiments. We will organize the presentation around the learning task, concrete operations, and evaluation targets, with the terminology and theoretical quantities explained directly in the main text.

State what the learner observes and does. In the theoretical preference-learning task, the learner knows the comparison features and prior, while the true reward parameter is unknown. Each round provides a prompt, a finite candidate set, and a reference chosen before current reward sampling. The learner samples reward functions, scores each candidate’s reward gap relative to the reference, selects a comparison, observes its binary preference label, and updates the posterior for the next round.

Give operational definitions at first use.

TermExplanation in the main text
Mean-plus-dispersion selectorScore a candidate using the mean plus a weighted standard deviation of its sampled reward evaluations.
CouplingSpecify how reward samples are drawn jointly, determining their dependence while preserving each sample’s prescribed distribution. IID sampling is the independent case.
Matched projection workCount one vocabulary projection of one token representation under one ensemble member as one unit, and match the total number of these operations.
Retained ensembleCarry the same set of 20 reward-model parameter vectors across rounds and update them after receiving new preference labels.
Informative challengerA candidate with slightly lower prior-mean reward than the high-mean alternative, but greater sensitivity to an uncertain parameter direction; it is identified using known features before feedback.
Encoder/orientation cellOne of 12 conditions crossing three frozen encoder sizes (1.5B, 3B, 7B) with four prior settings. The four settings select the more certain of two parameter coordinates and the sign of its prior mean.

Each study will also specify the scored object, SD coefficient, and SD convention.

Explain the assumptions and their roles. The theorem uses known bounded linear comparison features, rewards in [0,1][0,1], and a log-concave prior on a compact convex parameter domain. Known features define the likelihood used for posterior updating; bounded reward gaps keep logistic probabilities sensitive to changes in the gap; and log-concavity supports a sharper candidate-complexity constant, with an arbitrary-prior variant also provided. Each round uses a fresh pair of draws: conditional on the observed history and current inputs, the pair is independent of the unknown true parameter, while dependence within the pair is allowed.

We will explain the regret-bound quantities alongside the theorem:

  • ΓT\Gamma_T bounds the total information about the reward parameter available from TT rounds of preference feedback.
  • λ\lambda relates uncertainty in the queried reward gap to the information carried by its binary label.
  • CKC_K accounts for maximization over at most KK candidates.
  • The accumulated Wasserstein term accounts for approximation error between the sampling distribution and the current posterior.

Make each experiment’s question explicit in the main text. The learning workflow will precede Algorithm 1, and terminology will be explained at first use. Each experiment will open by stating what changes, what is held fixed, and what its endpoint measures: query regret evaluates the quality of queried responses, predictive NLL evaluates preference prediction, selector KL evaluates approximation of a specified selection rule, and F1/EM evaluate final-answer quality. The main text will contain these explanations needed to interpret the method and results, with proofs and detailed implementation in the appendices.

AuthorsOpenReview ↗

Rebuttal for W2-Part II

Thank you for asking how posterior theory, fixed neural reward models, and agentic training connect. The common question is how finite reward evaluations determine a selection, and how that selection affects subsequent feedback or training data. The paper analyzes both the learning process and the numerical implementation of selection, with experiments addressing these respective questions.

ComponentReward evaluationsUpdate and evaluation target
Posterior preference learningDifferent posterior reward functions evaluated on the same candidateThe selected comparison’s label updates the posterior; Section 3 analyzes query regret
Fixed neural selectorDropout reward predictions under a fitted modelHold the model fixed to study the finite-sampling precision considered in Section 4
Additional neural closed-loop studiesStochastic predictions from the currently trained reward headSelected label batches update the head, which then guides subsequent queries
Additional agentic group-filtering controlsRealized rewards of generated trajectories for the same promptRetained complete prompt groups enter GRPO policy updates

The fixed-model experiment directly addresses selection precision. Section 4 fixes the reward law and target selection rule to study how finite sampling affects implementation accuracy. Holding the fitted model fixed in Section 5.2 therefore isolates the effect of sampling design on selection error. Across four reward fits, forward KL to the reference selector decreases by 24.60%–26.16%, using 20 reward evaluations per candidate. This experiment tests the numerical role of finite reward evaluation.

Recorded human preferences provide an actual acquisition-and-update test. Our additional OpenAssistant replication fixes the data, two-dimensional representation, learner, and 512-query budget. Four methods run with 24 new seeds, yielding 96 complete learning trajectories. Each query reveals only the selected comparison’s recorded label, followed by a learner update before the next selection. The prespecified primary endpoint is normalized NLL area under the learning curve, where lower is better.

SamplingMean NLL AULCOriginal max2 OLEOLE − mean, 95% CIHolm p
IID0.6841950.683535−0.000660 [−0.000840, −0.000480]3.15×10−73.15\times10^{-7}
Coupled0.6842380.683525−0.000713 [−0.000873, −0.000553]1.39×10−81.39\times10^{-8}

These 24 new seeds are analyzed separately from the earlier eight. With the sampler held fixed, both comparisons show lower predictive loss over the learning trajectory under optimistic scoring.

Trainable reward heads further connect selection to repeated updates. The helpful-online experiment uses 128 paired seeds, 896 labels per method, and 21 fixed evaluation checkpoints. The reward head learns from acquired labels and subsequently guides query selection. Additional exploratory comparisons, holding the sampler fixed, give:

SamplingFinal NLL reduction from optimism (mean − optimistic)Adjusted paired-seed 95% CI
IID0.008437[0.006737, 0.010138]
Balanced0.010201[0.008415, 0.011986]

These exploratory contrasts are adjusted within a four-comparison family. They directly test how optimistic scoring affects subsequent prediction when the reward head is continually updated.

Agentic OLE applies the selection principle to policy training. In the additional group-filtering experiments, the mean-plus-SD score summarizes realized rewards of trajectories generated for the same prompt and determines which complete groups enter GRPO updates. Here, variation comes from generated trajectory outcomes; in preference acquisition, it comes from different reward functions evaluating the same candidate. The posterior regret guarantee concerns the preference model in Section 3, while the agentic extension is evaluated through matched mean-only and random filtering controls, detailed in W2c.

We will include this mapping in the task presentation and at the start of the experiments, making each study’s scored object, update process, and evidential role explicit.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.