acceptodds
Under review as a conference paper at ICLR 2027

Active Preference Querying for Offline-to-Online Reinforcement Learning

Abstract

Offline-to-online reinforcement learning usually assumes scalar rewards during online fine-tuning. We study offline-to-online preference-based RL, where an agent starts from offline preference data and can improve online only through a limited budget of pairwise preference queries, so deciding which comparisons to label is central. We propose an active preference-query rule that selects comparisons that are ambiguous to a reward ensemble and informative for the Bradley-Terry reward estimator, and APOLLO, an algorithm that applies this rule to long-horizon control with offline initialization and a flow-based, action-chunked policy. Our analysis gives a finite-sample bound on the preference-prediction error that holds for adaptively chosen queries; the query rule enters only through how well the labeled data cover the policy's comparisons, so its guarantee improves on random querying when the rule improves this coverage. On eight OGBench manipulation tasks, APOLLO outperforms five preference-based and behavioral-cloning baselines. When only the query rule is changed, our rule achieves the best average and worst-case success among six rules.

Primary area:
reinforcement learning
Keywords:
Active Learning, Preference Learning, Offline-to-online, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ESB1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents APOLLO, a method for offline-to-online preference-based RL. The agent starts from offline data with preference labels. During online training it gets no reward and can only ask a limited number of questions of the form "which of these two clips is better?". The main contribution is the rule for choosing which pairs to ask about. It prefers pairs on which an ensemble of reward models is uncertain, and adds a Fisher-information term that favours pairs that are informative for the reward model and unlike earlier queries. The rule is combined with an existing flow-based, action-chunked policy. The paper also gives a bound on the reward model's preference error that holds when queries are chosen adaptively.

Critical strengths
  1. The problem is well motivated and the idea is simple. When there is no reward online, deciding which pairs to label is the main lever, and the rule only needs a differentiable reward ensemble, so it can be combined with other policy learners. With the flow-based actor, the full system improves from offline to online on seven of eight sparse, long-horizon tasks, while the baselines stay much lower (Table 1).

  2. The theory allows each query to depend on earlier labels. This is what happens in active querying, and the usual bounds that assume independent samples do not cover it. The paper handles this with a martingale argument, and it is clear that its comparison with random querying is only between upper bounds.

Critical weaknesses
  1. It is not yet clear that this query rule is better than simpler ones. Without DOUBLE-TASK1, the full rule is about the same as No-IG and Disagreement, and there is no U^IG-only variant. Theorem 1 holds for any query rule, so the theory does not settle this either.

  2. The setting relies on strong assumptions. All tasks are OGBench cube manipulation, labels are simulated, and the method needs 10,000 offline preference labels, about ten times the online budget, to start well (Fig. 5, Appendix D.12). It is unclear how it would work in a new domain or without a cheap, good offline start.

Questions for authors
  1. Could you run Random, Disagreement, No-IG and the full rule on the other four tasks, and add a rule that uses only U^IG? If the full rule still helps there, I would raise my score.
  2. On DOUBLE-TASK1, Random collapses with the flow policy but not with the Gaussian one. Appendix D.13 also suggests that the policy exploits errors in the learned reward during offline training. Does the same happen online, and does the query rule reduce it? Tracking the reward model's accuracy on new pairs during training would show this.
Reviewer_ErfmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes APOLLO for offline-to-online preference-based RL. It combines ensemble uncertainty and a Fisher-information-based score to select online preference queries. The acquired labels update the reward model, which provides rewards for policy and critic updates. The theory bounds the preference prediction error. APOLLO outperforms the compared baselines on eight OGBench tasks, and ablations show that the combined query score achieves the highest average success among the tested rules.

Critical strengths
  1. The paper addresses learning from offline preferences with a limited online feedback budget.
  2. The experiments show strong performance, and the ablations support the effectiveness of the query selection rule.
Critical weaknesses
  1. Entropy-based selection and ensemble disagreement have already been used in PEBBLE. The main addition is an information-gain term, and the methodological novelty could be further clarified.
  2. The theory links better coverage to a tighter preference prediction error bound, but does not establish that the proposed selector improves coverage.
Questions for authors
  1. Would initializing the information matrix (\Lambda) using offline preference data help avoid querying comparisons already well covered by that data?
  2. Can you show whether adding the information-gain term actually reduces redundant comparisons? This would help clarify its benefit beyond uncertainty-based selection.
  3. The paper contains several unresolved citation markers, such as “?” on line 322.
Reviewer_6FEDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies offline-to-online preference-based RL, where an agent initialized from offline preferences can improve online only through a limited budget of pairwise queries. The main contribution is an active query rule that adds ensemble uncertainty (entropy plus disagreement) to an ambiguity-weighted Fisher information-gain term and labels the top K = 4 of 32 candidate pairs per round. Theorem 1 bounds the preference-prediction error for adaptively chosen queries; the rule enters only through a coverage constant C_cov(U), and the authors don’t prove that the rule lowers it. APOLLO puts the rule into an offline-initialized, flow-based, action-chunked actor-critic and beats five baselines on eight OGBench cube tasks, and in a controlled comparison of six acquisition rules on four tasks the proposed score has the best mean and worst-task success.

Critical strengths

(1) The theory is honest about what it gives. The bound holds for adaptively chosen queries, the rule only enters through C_cov(U), and Appendix D.7 shows a case where the rule helps (α = 0.6) and one where it hurts (ratio 5/3); Appendix A.6 even gives the exact condition for beating random querying.

(2) Table 2 is the right experiment: backbone, reward ensemble, candidate pool and budget stay fixed and only the acquisition rule changes. The appendix studies also report the bad cases, such as TRIPLE-TASK4 under 20% label flips.

Critical weaknesses

(1) The case that the rule beats simpler rules rests on one task. On DOUBLE-TASK1 the full score reaches 0.82 and every other rule is at most 0.40; on the other three tasks in Table 2 the means are 0.72 for the full score and 0.70 for both No-IG and Disagreement. On DOUBLE-TASK1, random querying drops from the 0.25 offline start (Table 1) to 0.01, while the Gaussian Uni-IVR backbone with random querying gets 0.43 (Table 6), so part of the gap may be an instability of the flow-based pipeline that the rule happens to avoid. K = 4 was also picked by task success on TRIPLE-TASK4 (Appendix D.3), one of the same four tasks, although Appendix D.13 rightly refuses to pick the offline checkpoint by success.

(2) The new part of U_t is neither isolated nor fully specified. Entropy and Disagreement come from PEBBLE/B-Pref, and gᵀΛ⁻¹g is the D-optimal criterion of Ji et al. (2025), so what is new is the ambiguity weight p̄(1−p̄) and the sum; Table 2 has no U^ens plus the unweighted term. The text doesn’t say how g and Λ_t^{-1} are computed for a neural reward ensemble, so the main contribution can’t be reimplemented. The theory doesn’t explain the gains either: at your label split (w ≈ 0.093), even the favorable example in Appendix D.7 gives only α ≈ 0.85, so Theorem 1 can’t account for the 0.82 vs 0.40 gap.

Questions for authors

Q1. What causes the collapse of random querying on DOUBLE-TASK1? The single-criterion rules have standard errors of 0.19–0.29 there, which looks like some seeds succeed and others collapse. Could you show per-seed results and run all six rules on DOUBLE-TASK2/3 and TRIPLE-TASK2/3, to see whether DOUBLE-TASK1 is typical?

Q2. Can you pick K on tasks outside Table 2, or rerun Table 2 with K fixed in advance?

Q3. Can you add U^ens plus the unweighted elliptical term to Table 2, so the effect of the ambiguity weight is visible?

Q4. How is U_t^IG computed? If the reward networks follow the MLP in Table 4, is g taken over the full parameter vector or something like the last layer, which ensemble member (or the mean) provides it, and how is Λ_t^{-1} kept up to date? The wall-clock overhead over No-IG would be good to report too.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.