acceptodds
Under review as a conference paper at ICLR 2027

Critic-Grounded Online Fine-Tuning with Active Preference Elicitation

Abstract

The offline-to-online paradigm in reinforcement learning deploys a pretrained actor-critic checkpoint without access to the offline dataset or the reward function that produced it. The data may be withheld for privacy or proprietary reasons, and the reward may have been handcrafted by experts. We study online fine-tuning in this regime, where the only available supervision is preference feedback: a labeler compares two short behavior segments, may abstain, and never provides a per-step reward. Two obstacles arise in this setting. First, a reward learned from preferences is identified only up to a scale and an offset, which the standard Bradley-Terry likelihood cannot pin down, whereas the pretrained critic holds values in the units of the offline reward. Writing an arbitrarily placed reward into the replay buffer corrupts those values and destabilizes fine-tuning. We design a critic gauge, which recovers the reward implied by the frozen critic and regresses the learned reward onto it to fix the scale and offset. Second, preference labels are costly, and the labeler may abstain, so comparisons must be chosen to be both informative and answerable. We score each candidate by the expected reduction in posterior variance across the pool, under a Gaussian posterior over the reward model's parameters, weighted by a calibrated online estimate of the labeler's likelihood of answering. Our selection is sequential, folding each label into the posterior before the next pair is chosen so that redundant comparisons are never selected. On eight standard RL environments, our method closely matches WSRL, a baseline that shares our no-data-retention setting but observes the true online reward, and substantially outperforms PEBBLE, which learns from preferences like ours but starts without a pretrained checkpoint.

Primary area:
reinforcement learning
Keywords:
Offline-to-Online RL, Reinforcement Learning, Reward Learning, Active Sampling

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_mPZfOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses offline to online RL with a pre-trained value function and learning a reward function on the fly. The main proposal is the reward gauge which aligns the online reward with the offline reward basis which can be extracted from the pre-trained value function via the inverse Bellman operator. The paper further addresses active data selection by querying for the most informative and anwerable data pairs. They estimate information gain using Laplace approximation. Experiment are conducted in D4RL environments showing that the proposed method approaches WSRL with known online reward, significantly outperform no-gauge, and benefit from active selection.

Critical strengths
  • The paper is well written. The proposed method is sound and easy to understand.
  • The paper situated itself among prior work WSRL and PEBBLE very well, ran very reasonable ablations, and demonstrated expected effects.
Critical weaknesses
  • While reward gauge and the offline to online setting are clear contributions from this paper, I find the active selection an unclear contribution. There already exist a handful of work applying active selection which are also based on approximate Bayesian inference, e.g., [1, 2].

[1]. Poddar, S., Wan, Y., Ivison, H., Gupta, A., & Jaques, N. (2024). Personalizing reinforcement learning from human feedback with variational preference learning. [2]. Biyik, E., & Sadigh, D. (2018, October). Batch active preference-based learning of reward functions.

Questions for authors

NA

2 comments

Rebuttal to Reviewer mPZf

While reward gauge and the offline to online setting are clear contributions from this paper, I find the active selection an unclear contribution. There already exist a handful of work applying active selection which are also based on approximate Bayesian inference, e.g., [1, 2].

[1] Poddar et al. (2024). Personalizing reinforcement learning from human feedback with variational preference learning. [2] Biyik & Sadigh (2018). Batch active preference-based learning of reward functions.

The reviewer is right that active query selection with approximate Bayesian inference is already established in preference learning, and we do not claim it as new. Poddar et al. (2024) choose queries to identify a user's latent preference, and Biyik and Sadigh (2018) choose a diverse batch of queries at once for a reward that is linear in features. In both, every query receives a label. Our teacher may abstain, so we weight each candidate pair by a calibrated estimate of the probability that the teacher answers it, computed from the return gap implied by the frozen critic. Starting from a pretrained checkpoint is what makes this estimate available. We also score a pair by how much its label reduces the posterior variance of the reward over the whole pool. We will cite both papers and state these differences in Section 1.1.

Thank the authors for the response. I have no other questions.

Reviewer_EhqkOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates a training paradigm of offline-to-online reinforcement learning in which the offline dataset and original reward function are unavailable during online fine-tuning. Two issues are identified in this training regime: reward calibration, where a reward model learned from preferences may have a different scale and offset from the original reward, and preference efficiency, where preference queries are expensive and some comparisons may be unanswerable. To address these issues, the authors propose a new algorithm, ANCOR, that combines critic gauge, which fits an affine transformation of the preference-learned reward, and active selection, which selects trajectory pairs by maximizing expected posterior-variance reduction across a candidate pool. Experiments on eight D4RL environments show that ANCOR achieves performance generally close to that of WSRL, which has access to the online reward signal, and substantially outperforms PEBBLE, which learns from preferences without pretraining.

Critical strengths
  1. The paper focuses on an interesting and relatively underexplored setting of offline-to-online RL, and I think this is meaningful for applications of offline-to-online RL that have privacy concerns.
  2. The results of the empirical study are promising, and the proposed method achieves performance close to or matching WSRL (baseline with access to true reward signal), showing the effectiveness of the proposed method. The authors also tried different pretrained checkpoints using different offline RL algorithms, showing the generality of the method.
Critical weaknesses
  1. The proposed calibration assumes that the pretrained critic is approximately Bellman-consistent with the true reward. However, offline-trained critics can have substantial estimation errors, particularly for OOD data. The implementation estimates the gauge using the first 5000 online transitions collected by the frozen policy and continues using that reference distribution throughout fine-tuning. It is unclear whether the learned calibration remains reliable as the policy visits states and actions that are increasingly different from those encountered during warm-up.
  2. The baseline methods (WSRL and PEBBLE) are relevant, but a simple baseline seems to be missing, which is to use a newly randomly initialized critic network, reuse the pretrained actor network, and fine-tune using the preference-learned reward.
Questions for authors

See Weaknesses

3 comments

Rebuttal to Reviewer Ehqk

1. The proposed calibration assumes that the pretrained critic is approximately Bellman-consistent with the true reward. However, offline-trained critics can have substantial estimation errors, particularly for OOD data. The implementation estimates the gauge using the first 5000 online transitions collected under the frozen policy and then continues to use that reference distribution throughout fine-tuning. It is unclear whether the learned calibration remains reliable as the policy visits states and actions that are increasingly different from those encountered during warm-up.

The reviewer is right that we fit the gauge on the warm-up transitions for the whole run. One way to handle the shift in state distribution is to refit the gauge on recent transitions. We observe that recalibration yields no significant improvement: it performs worse on most tasks and about the same on the others (Table R2).

The reason is that the gauge estimates a single global affine map, one scale and one offset, rather than a state-dependent correction. The scale relating the learned reward to the critic's units is not a property of the state distribution on which they are compared, so a fit from warm-up data continues to place the reward correctly as visitation shifts.

Table R2: Normalized score at 50k, 100k, 200k and 300k online steps (success rate on Pen and Door), mean ±\pm standard error over 10 seeds. Pen and Door are trained for 100k steps. Refit on recent data: at each gauge fit, the scale and offset are fit to the latest 5,000 transitions rather than the first 5,000 (warm-up) transitions; both use the frozen prior critic.

TaskMethod50k100k200k300k
Kitchen-mixedANCOR0.22±0.070.22 \pm 0.070.43±0.040.43 \pm 0.040.62±0.050.62 \pm 0.050.65±0.040.65 \pm 0.04
Refit on recent data0.06±0.030.06 \pm 0.030.12±0.030.12 \pm 0.030.04±0.030.04 \pm 0.030.11±0.040.11 \pm 0.04
Kitchen-partialANCOR0.32±0.040.32 \pm 0.040.45±0.060.45 \pm 0.060.62±0.050.62 \pm 0.050.72±0.050.72 \pm 0.05
Refit on recent data0.16±0.050.16 \pm 0.050.28±0.070.28 \pm 0.070.37±0.080.37 \pm 0.080.58±0.110.58 \pm 0.11
AntMaze-diverseANCOR0.54±0.100.54 \pm 0.100.77±0.090.77 \pm 0.090.91±0.020.91 \pm 0.020.92±0.020.92 \pm 0.02
Refit on recent data0.39±0.100.39 \pm 0.100.71±0.090.71 \pm 0.090.84±0.080.84 \pm 0.080.90±0.040.90 \pm 0.04
AntMaze-playANCOR0.68±0.090.68 \pm 0.090.86±0.020.86 \pm 0.020.92±0.020.92 \pm 0.020.97±0.010.97 \pm 0.01
Refit on recent data0.56±0.110.56 \pm 0.110.65±0.080.65 \pm 0.080.79±0.100.79 \pm 0.100.96±0.020.96 \pm 0.02
PenANCOR0.78±0.050.78 \pm 0.050.81±0.030.81 \pm 0.03––
Refit on recent data0.78±0.030.78 \pm 0.030.88±0.030.88 \pm 0.03––
DoorANCOR0.35±0.080.35 \pm 0.080.78±0.090.78 \pm 0.09––
Refit on recent data0.32±0.090.32 \pm 0.090.89±0.040.89 \pm 0.04––
HalfcheetahANCOR0.52±0.020.52 \pm 0.020.61±0.020.61 \pm 0.020.69±0.020.69 \pm 0.020.74±0.020.74 \pm 0.02
Refit on recent data0.05±0.050.05 \pm 0.050.29±0.090.29 \pm 0.090.45±0.100.45 \pm 0.100.43±0.120.43 \pm 0.12
WalkerANCOR0.50±0.110.50 \pm 0.110.63±0.110.63 \pm 0.110.83±0.110.83 \pm 0.110.97±0.070.97 \pm 0.07
Refit on recent data0.06±0.030.06 \pm 0.030.12±0.050.12 \pm 0.050.29±0.120.29 \pm 0.120.36±0.120.36 \pm 0.12

2. The baseline methods (WSRL and PEBBLE) are relevant, but a simple baseline seems to be missing, which is to use a newly randomly initialized critic network, reuse the pretrained actor network, and fine-tune using the preference-learned reward.

We ran this baseline, and Table R3 reports it as Fresh critic. With a fresh critic, the final score is less than half of ANCOR's on AntMaze and Adroit and lower on HalfCheetah. On Kitchen and Walker, it matches ANCOR by the end of training. We will add this baseline to the main results of the paper.

Table R3: Normalized score at 50k, 100k, 200k and 300k online steps (success rate on Pen and Door), mean ±\pm standard error over 10 seeds. Pen and Door are trained for 100k steps. WSRL is given the hand-written reward; PEBBLE learns from scratch without a prior.

TaskMethod50k100k200k300k
Kitchen-mixedANCOR0.22±0.070.22 \pm 0.070.43±0.040.43 \pm 0.040.62±0.050.62 \pm 0.050.65±0.040.65 \pm 0.04
Fresh critic0.19±0.040.19 \pm 0.040.31±0.060.31 \pm 0.060.57±0.060.57 \pm 0.060.67±0.040.67 \pm 0.04
PEBBLE0.00±0.000.00 \pm 0.000.03±0.030.03 \pm 0.030.05±0.030.05 \pm 0.030.06±0.030.06 \pm 0.03
WSRL0.37±0.030.37 \pm 0.030.53±0.050.53 \pm 0.050.57±0.030.57 \pm 0.030.58±0.050.58 \pm 0.05
Kitchen-partialANCOR0.32±0.040.32 \pm 0.040.45±0.060.45 \pm 0.060.62±0.050.62 \pm 0.050.72±0.050.72 \pm 0.05
Fresh critic0.28±0.040.28 \pm 0.040.45±0.030.45 \pm 0.030.63±0.030.63 \pm 0.030.71±0.020.71 \pm 0.02
PEBBLE0.03±0.030.03 \pm 0.030.00±0.000.00 \pm 0.000.05±0.030.05 \pm 0.030.03±0.020.03 \pm 0.02
WSRL0.45±0.050.45 \pm 0.050.66±0.050.66 \pm 0.050.71±0.060.71 \pm 0.060.79±0.070.79 \pm 0.07
AntMaze-diverseANCOR0.54±0.100.54 \pm 0.100.77±0.090.77 \pm 0.090.91±0.020.91 \pm 0.020.92±0.020.92 \pm 0.02
Fresh critic0.00±0.000.00 \pm 0.000.02±0.020.02 \pm 0.020.42±0.110.42 \pm 0.110.44±0.100.44 \pm 0.10
PEBBLE0.00±0.000.00 \pm 0.000.00±0.000.00 \pm 0.000.00±0.000.00 \pm 0.000.00±0.000.00 \pm 0.00
WSRL0.75±0.030.75 \pm 0.030.87±0.020.87 \pm 0.020.96±0.010.96 \pm 0.010.95±0.010.95 \pm 0.01
AntMaze-playANCOR0.68±0.090.68 \pm 0.090.86±0.020.86 \pm 0.020.92±0.020.92 \pm 0.020.97±0.010.97 \pm 0.01
Fresh critic0.01±0.010.01 \pm 0.010.02±0.020.02 \pm 0.020.18±0.080.18 \pm 0.080.36±0.140.36 \pm 0.14
PEBBLE0.00±0.000.00 \pm 0.000.00±0.000.00 \pm 0.000.00±0.000.00 \pm 0.000.00±0.000.00 \pm 0.00
WSRL0.72±0.050.72 \pm 0.050.91±0.020.91 \pm 0.020.97±0.010.97 \pm 0.010.97±0.020.97 \pm 0.02
PenANCOR0.78±0.050.78 \pm 0.050.81±0.030.81 \pm 0.03––
Fresh critic0.19±0.050.19 \pm 0.050.24±0.090.24 \pm 0.09––
PEBBLE0.08±0.020.08 \pm 0.020.10±0.020.10 \pm 0.02––
WSRL0.81±0.020.81 \pm 0.020.92±0.020.92 \pm 0.02––
DoorANCOR0.35±0.080.35 \pm 0.080.78±0.090.78 \pm 0.09––
Fresh critic0.00±0.000.00 \pm 0.000.04±0.030.04 \pm 0.03––
PEBBLE0.00±0.000.00 \pm 0.000.00±0.000.00 \pm 0.00––
WSRL0.32±0.080.32 \pm 0.080.95±0.020.95 \pm 0.02––
HalfcheetahANCOR0.52±0.020.52 \pm 0.020.61±0.020.61 \pm 0.020.69±0.020.69 \pm 0.020.74±0.020.74 \pm 0.02
Fresh critic0.42±0.050.42 \pm 0.050.52±0.030.52 \pm 0.030.61±0.020.61 \pm 0.020.67±0.020.67 \pm 0.02
PEBBLE0.01±0.010.01 \pm 0.010.03±0.010.03 \pm 0.010.16±0.030.16 \pm 0.030.22±0.040.22 \pm 0.04
WSRL0.70±0.010.70 \pm 0.010.75±0.010.75 \pm 0.010.82±0.010.82 \pm 0.010.86±0.010.86 \pm 0.01
WalkerANCOR0.50±0.110.50 \pm 0.110.63±0.110.63 \pm 0.110.83±0.110.83 \pm 0.110.97±0.070.97 \pm 0.07
Fresh critic0.12±0.020.12 \pm 0.020.50±0.090.50 \pm 0.090.79±0.090.79 \pm 0.091.03±0.031.03 \pm 0.03
PEBBLE0.03±0.010.03 \pm 0.010.05±0.010.05 \pm 0.010.14±0.040.14 \pm 0.040.38±0.100.38 \pm 0.10
WSRL0.75±0.060.75 \pm 0.060.84±0.070.84 \pm 0.070.88±0.080.88 \pm 0.081.09±0.031.09 \pm 0.03

Thanks for the detailed rebuttal and clarification, which addressed and resolved my concerns. I will update the score accordingly.

Reviewer_AxX2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates a valuable reinforcement learning problem: how to perform stable online fine-tuning of a pre-trained Actor-Critic model using only a small amount of preference feedback, in a setting where neither the original offline training data nor the ground-truth reward function is accessible.

Critical strengths
  1. The research problem holds practical significance. The paper addresses an "Offline-to-Online" reinforcement learning (RL) scenario subject to realistic constraints: both the original training data and the reward function are inaccessible, with only a pre-trained model and limited preference feedback available. Compared to traditional preference-based RL, this setting places greater emphasis on preserving the capabilities of the existing policy while enabling its continuous improvement.

  2. The experimental comparisons and ablation analyses are quite comprehensive. The authors conduct experiments across eight diverse types of reinforcement learning tasks, analyzing the impacts of reward calibration, active querying, various offline pre-training algorithms, and preference budgets. The main results are based on 10 random seeds, while the active querying experiments utilize 20 random seeds, providing a reasonable degree of statistical support.

Critical weaknesses
  1. The core method relies on a frozen Critic to provide relatively reliable Bellman-backpropagated rewards. However, an offline Critic is susceptible to issues such as function approximation errors, conservative estimation bias, and challenges with out-of-distribution generalization. While the authors employ a scaling factor to mitigate estimation uncertainty, it remains unclear whether reward calibration remains reliable when the Critic’s errors exhibit systematic bias.

  2. The paper provides theoretical guarantees for greedy optimization based on approximate submodularity—a valuable contribution. However, the appendix clarifies that these guarantees assume a fixed batch of targets and a fixed query information coefficient, while treating "abstentions" (decisions not to provide an answer) as uninformative events. In contrast, the actual algorithm updates the posterior mean each time a label is obtained and subsequently recalculates the scores for future queries. Consequently, these theoretical guarantees cannot be directly extended to the full, adaptive sequential algorithm.

Questions for authors

see Weaknesses

2 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer AxX2

1. The core method relies on a frozen Critic to provide relatively reliable Bellman-backpropagated rewards. However, an offline Critic is susceptible to issues such as function approximation errors, conservative estimation bias, and challenges with out-of-distribution generalization. While the authors employ a scaling factor to mitigate estimation uncertainty, it remains unclear whether reward calibration remains reliable when the Critic's errors exhibit systematic bias.

We measured the critic's error across all 8 tasks, with 10 seeds per task. On the 5,000 warm-up transitions used to fit the gauge, the reward implied by the frozen critic, Qˉ(s,a)−γVˉ(s′)\bar{Q}(s,a)-\gamma\bar{V}(s'), overestimates the true reward on 79 of the 80 seeds. Across tasks, the median overestimate ranges from 0.3 to 1.9 reward units, and the median per-transition correlation with the true reward ranges from 0.0 to 0.3. The critics are biased and noisy, as the reviewer describes. Our results in Figures 1- 4 already show that the proposed calibration (CriticGauge) helps overcome this systematic bias.

The calibration also holds for critics trained with different forms of conservatism. We fine-tuned Pen from priors trained by four offline algorithms, and with each of them, the gauge is already ahead at 25k online steps, more than doubles the success rate at 50k, and is still about two to three times as high at 100k (Table R1).

Table R1: Pen: success rate at 25k, 50k and 100k online steps with and without the critic gauge, for priors trained by four offline algorithms (10 seeds per arm).

PriorWith gauge (25k / 50k / 100k)Without gauge (25k / 50k / 100k)
CQL0.470 / 0.720 / 0.8500.210 / 0.120 / 0.295
Cal-QL0.435 / 0.630 / 0.7600.240 / 0.265 / 0.300
IQL0.290 / 0.560 / 0.6900.175 / 0.235 / 0.355
AWAC0.320 / 0.590 / 0.8050.190 / 0.140 / 0.300

The reason is that the gauge takes only two parameters from the critic, a scale α\alpha and an offset β\beta, which are fit by regression over the 5,000 transitions. Critic bias enters this regression in two ways. A bias that is roughly constant over the warm-up transitions is absorbed by the offset β\beta. That places the reward in the critic's own units, which is what fine-tuning needs. A bias that varies across transitions acts as noise in the regression. Errors unrelated to the learned reward average out over the 5,000 transitions, and the shrinkage factor ω\omega lowers the scale when the fit is noisy. Only a critic error that is correlated with the learned reward itself can tilt the scale.

2. The paper provides theoretical guarantees for greedy optimization based on approximate submodularity—a valuable contribution. However, the appendix clarifies that these guarantees assume a fixed batch of targets and a fixed query information coefficient, while treating "abstentions" (decisions not to provide an answer) as uninformative events. In contrast, the actual algorithm updates the posterior mean each time a label is obtained and subsequently recalculates the scores for future queries. Consequently, these theoretical guarantees cannot be directly extended to the full, adaptive sequential algorithm.

Yes, the guarantee holds for the batch objective, and the deployed sequential rule is a heuristic refinement that performs on par with the batch objective in six environments and is slightly better on two environments (Figure C.1). The batch variant can therefore be used with the full guarantee at very little measured cost. We have already mentioned this in the appendix and will put a remark in the main text.

Reviewer_AxX2OpenReview ↗

response to author

The author addressed all my questions, so I’ve decided to increase my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.