- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors propose Optimistic Learning with Ensembles (OLE). It focuses on the intersection of optimistic sampling, active preference learning, and reward ensembles. Specifically, it uses finite reward ensembles to select which preference comparisons or training experiences should be acquired. The experiments show improvement on the neural reward model and agentic setting.
Critical strengths
-
The authors identify an underexplored distinction in the intersection between reward ensemble and data selection.
-
The motivation is theoretically grounded, and the main theoretical treatment is careful. The two-round construction makes the exploration tradeoff concrete.
Critical weaknesses
-
The paper currently feels like several related papers placed under one umbrella, and the connection between the theoretical OLE object and the later experiments is weak. Sections 2–3 use posterior reward-function draws to evaluate the same candidates, but Section 5.2 uses no learning or posterior update. Moreover, agentic OLE only uses variation comes from realized rewards across generated trajectories with no posterior preference-learning loop present either.
-
The strongest positive preference-learning result appears highly dependent on a deliberately selected “informative challenger” geometry. With OpenAssistant geometry, the optimism seems not helpful, even when the labels are replaced with correctly specified synthetic BTL feedback, mean selection has lower regret. Thus, OLE seems not generally preferable for preference acquisition.
-
The Agentic OLE experiments contain major attribution and reproducibility problems. The appendix states that it is unresolved whether the rejection ablation used trajectory-level selection or prompt-group selection, so the experiment cannot identify which selection rule caused the gain. Given that this is the portion of the paper that most directly supports the relevance to modern LLM agents, I think these issues need to be resolved rather than only documented as limitations.
-
The writing flow needs substantial improvement:
-
The paper should give an intuitive understanding of this optimistic reward ensemble. Even after finishing reading the introduction and the beginning of Section 2, it is not clearly presented why this optimism is important in reward ensemble and pair selection.
-
A formulation of the task focus is preferred before Section 2. Currently, the method directly begins without clearly presenting the problem definition.
-
Questions for authors
-
What exactly is the common mathematical object linking Agentic OLE to Sections 2–4? Can the trajectory-selection rule be written as an instance of the same finite-ensemble selection objective, or should Agentic OLE instead be presented as a heuristic inspired by OLE?
-
Which rejection implementation produced the reported Agentic OLE numbers? The appendix currently says this is unresolved. This seems essential to resolve before publication. A matched-volume mean-only rejection baseline in the agentic experiments would be helpful. This would directly test whether the dispersion/optimism component contributes beyond filtering low-reward trajectories.
-
Can the paper provide a practical statistic, computed before collecting labels, that predicts whether a candidate set resembles the “informative challenger” regime in which optimism is expected to repay its cost? This would turn the negative natural-geometry results into a stronger scientific conclusion.
-
Is there a realistic sequential learning experiment in which balanced sampling's reduction in selector approximation error produces a measurable downstream learning gain? This would substantially strengthen the bridge between Section 4 and the broader OLE motivation.
9 comments
Rebuttal for W2
The additional experiments extend the evidence to OpenAssistant and test whether favorable conditions for optimism can be identified before new feedback. This complements the exact two-round analysis in Section 3.2: exploration reduces cumulative regret when the future decision benefit of additional information exceeds its immediate query cost.
1. Predictive gains on OpenAssistant human preferences. The new study uses frozen Qwen3-1.7B features and two-dimensional PCA fitted only on training data. Four methods run with 24 new paired seeds and 512 queries per learning trajectory. Each method acquires labels for its own selected comparisons and updates its reward learner. The prespecified primary endpoint is normalized NLL area under the learning curve:
| Sampler | Mean | Max2 | Max2 minus mean, 95% CI | Holm p |
|---|---|---|---|---|
| IID | 0.684195 | 0.683535 | −0.000660 [−0.000840, −0.000480] | |
| Coupled | 0.684238 | 0.683525 | −0.000713 [−0.000873, −0.000553] |
These 24 new seeds are analyzed separately from the earlier eight. Both samplers support a small predictive benefit from optimistic scoring, providing direct evidence beyond the informative-challenger construction. The endpoint here is predictive quality.
2. Sampling and exploration strength on OpenAssistant geometry. In a separate study with 128 BTL episodes and 512 queries per trajectory, holding the score fixed and replacing IID with coupled sampling reduces mean-rule regret from 10.0612 to 8.1639 (18.86%) and max2 regret from 10.2372 to 9.3991 (8.19%). These comparisons directly test sampling dependence. Within this study, coupled mean retains lower regret than coupled max2: 8.1639 versus 9.3991.
Independent confirmation also supports adjusting exploration strength. After freezing the development-selected dispersion coefficient of 0.25, evaluation on 128 new BTL episodes reduces coupled-rule regret from 9.0738 under the original coefficient of 1 to 8.0621, an 11.15% reduction. The paired difference is −1.0116, with 95% CI [−1.5193, −0.5040] and Holm p = 0.000793 across seven prespecified comparisons. This identifies the benefit of adapting exploration strength to the task conditions.
3. Identifying favorable conditions before new feedback. We freeze a posterior-forecast diagnostic and threshold on 48 development environments, then evaluate them on 160 independent simulated-BTL confirmation environments, equally divided between natural-text and informative-challenger geometries. This separate study uses posterior-moment scores with softmax selection at temperature 0.05.
The diagnostic achieves AUROC 0.7319, with 95% CI [0.6324, 0.8242]. Before confirmation feedback, the frozen threshold identifies 98 environments as favorable. Within these 98 environments, optimistic selection reduces cumulative regret relative to mean selection by 6.1335 on average, with multiplicity-adjusted 95% CI [4.5477, 7.7192]. This evaluates the benefit within a prospectively identified favorable subset. Our response to Q3 provides the full diagnostic protocol and results.
Rebuttal for W3
We have added controlled experiments with an explicitly specified filter, matched retained volume, and paired training runs. These experiments will provide the primary evidence for dispersion-specific claims in the revision. The available historical records do not yet uniquely identify which implementation produced the original rejection-ablation checkpoints; the new experiments execute the following identified operation.
Explicit group-level filtering. Each prompt produces four trajectories with rewards . We compute
\qquad s_b=\sqrt{\frac14\sum_{i=1}^{4}(r_{b,i}-\mu_b)^2}.$$ Mean+SD ranks prompt groups by $\mu_b+s_b$, mean-only ranks them by $\mu_b$, and random retention uses a random ranking. All four trajectories in each selected group are retained. Within each study, filtered methods match initialization conditions, action interfaces, generated-trajectory counts, retained-trajectory counts, and update counts. The mean+SD versus mean-only comparison therefore directly tests the contribution of adding dispersion to the ranking score. **Results from 512 steps and eight paired seeds.** Each step generates 16 groups and retains eight or four, corresponding to 50% or 25% retention. Each arm and seed generates 32,768 trajectories and retains 16,384 or 8,192, respectively. Against mean-only at the same retained volume: | Retention / evaluation | Mean+SD minus mean F1 (pp) | Adjusted paired-seed 95% CI | |---|---:|---| | 50% / primary evaluation | +3.21 | [0.60, 5.82] | | 25% / primary evaluation | +7.03 | [5.47, 8.59] | | 50% / 2,048 additional questions | +3.29 | [2.13, 4.45] | | 25% / 2,048 additional questions | +8.29 | [6.71, 9.87] | The primary and additional-question evaluations are adjusted within separate four-comparison families. The additional questions evaluate the same trained models. These comparisons support a final-answer F1 benefit from the dispersion term over mean-only filtering in this setting. **Other training settings.** In the 100-update study with six paired seeds, the mean+SD-minus-mean-only F1 difference is −0.0368 pp, with 95% CI [−0.7201, 0.6250]. In the 50-update study with three matched warm starts, the corresponding difference is −0.2423 pp, with 95% CI [−0.9960, 0.4864]. The latter study also improves answer validity over random retention by 0.9766 pp, with 95% CI [0.0326, 2.0508], identifying an answer-format benefit. These studies have different reward, normalization, and initialization settings and will be reported separately. **Reproducibility documentation.** For the new experiments, the revised appendix will provide filtering pseudocode and associate each run configuration, seed, checkpoint, and per-seed result. It will also specify reward and advantage definitions, generation and retention counts, and evaluation protocols, so that each filtering claim is tied to an identified executed procedure.
Rebuttal for W1
The common question is how to select data for learning under a finite reward-evaluation budget, and how accurately that selection can be implemented. The components examine the learning consequences of selection, the implementation precision of finite ensembles, and an application to selecting policy-training experience. They share the scoring form
Fa(v)=mean(v)+asd(v),
with the source of v and the selected object specified separately:
Why the fixed-model neural study connects to the theory. Section 4 fixes the reward law and target selection rule, then analyzes selection error from finite reward evaluations. Section 5.2 directly tests this question: at the same budget of 20 reward evaluations, does changing the sampling arrangement implement the target rule more accurately? Holding the model fixed isolates this sampling error. Its theoretical counterpart is the selector-precision analysis in Section 4. The additional closed-loop studies then examine learning outcomes when selection feeds into actual model updates.
Additional experiments complete the selection–feedback–update loop. The new neural studies select queries using the current reward head, obtain their preference labels, update the head, and use it for subsequent selection. In particular, the ActiveUltraFeedback experiments update after each batch of 32 labels, with each sampled mask shared across candidates of the same prompt.
On helpful-online, using recorded human preferences, each method receives 896 labels across 128 paired seeds. Holding the sampler fixed, optimistic scoring reduces final NLL relative to mean scoring:
These exploratory contrasts were evaluated after the primary analysis and corrected as a separate four-comparison family. They directly evaluate how the selection rule affects predictive quality through actual reward-head updates at a matched label budget.
Connection to the agentic application. In the additional group-filtering experiments, trajectory rewards are available before retention, and the mean-plus-dispersion score determines which prompt groups enter policy updates. The updated policy generates the next round of trajectories, linking experience selection to policy learning. Here, dispersion describes realized reward variation across trajectories. Theorem 3.1 provides the guarantee for posterior sampling under the bounded linear-logistic model; matched policy-training controls evaluate the learning effects of the agentic application.